Jul 20 13:14:34.225678 ip-10-0-142-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 13:14:34.225692 ip-10-0-142-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 13:14:34.225701 ip-10-0-142-97 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 13:14:34.226041 ip-10-0-142-97 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 13:14:44.335976 ip-10-0-142-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 13:14:44.335999 ip-10-0-142-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fda7c155142c4fd1a6b3e32626487544 -- Jul 20 13:16:39.989434 ip-10-0-142-97 systemd[1]: Starting Kubernetes Kubelet... Jul 20 13:16:41.647951 ip-10-0-142-97 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:41.647951 ip-10-0-142-97 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 13:16:41.647951 ip-10-0-142-97 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:41.647951 ip-10-0-142-97 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 13:16:41.647951 ip-10-0-142-97 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:41.651729 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.651632 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 13:16:41.656093 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656074 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.656093 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656092 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.656093 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656096 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656100 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656103 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656107 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656109 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656112 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656116 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656119 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656121 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656124 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656127 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656129 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656132 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656135 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656138 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656142 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656144 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656147 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656151 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.656187 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656155 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656158 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656161 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656163 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656166 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656169 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656171 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656174 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656176 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656179 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656182 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656184 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656187 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656190 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656193 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656196 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656198 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656201 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656204 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.656714 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656208 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656211 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656214 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656216 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656219 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656222 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656224 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656227 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656229 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656232 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656235 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656237 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656240 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656242 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656245 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656248 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656251 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656253 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656256 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656258 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.657484 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656261 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656263 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656267 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656270 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656272 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656275 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656277 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656280 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656283 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656285 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656288 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656291 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656294 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656296 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656299 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656301 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656304 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656306 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656308 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656311 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.657999 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656313 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656316 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656319 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656321 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656324 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656326 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656329 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656333 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656336 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656339 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656341 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656344 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656347 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656349 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656370 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656376 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656381 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656386 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.656390 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.658510 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658303 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658316 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658319 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658323 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658326 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658330 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658333 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658335 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658338 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658341 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658344 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658347 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658365 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658370 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658373 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658377 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658381 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658386 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658388 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658391 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.658966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658394 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658396 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658399 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658402 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658406 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658411 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658416 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658420 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658422 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658425 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658428 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658431 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658434 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658437 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658440 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658442 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658445 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658448 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658451 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.659472 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658453 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658456 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658458 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658461 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658463 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658467 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658469 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658471 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658474 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658476 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658479 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658481 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658484 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658486 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658489 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658492 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658494 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658498 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658502 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658504 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.659922 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658507 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658509 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658512 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658515 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658517 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658520 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658524 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658528 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658531 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658534 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658537 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658539 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658542 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658544 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658547 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658550 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658552 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658555 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658557 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658560 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.660420 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658562 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658565 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658567 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658569 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658574 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658578 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658581 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658583 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658586 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658589 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658592 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658595 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658598 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658600 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658603 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658605 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658608 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658611 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658613 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.660926 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.658616 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659011 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659016 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659019 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659021 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659024 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659027 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659029 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659032 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659035 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659037 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659040 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659043 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659046 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659048 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659051 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659054 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659057 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659059 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.661410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659062 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659065 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659067 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659070 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659074 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659077 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659079 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659082 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659084 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659087 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659089 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659092 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659094 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659097 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659100 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659109 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659112 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659114 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659117 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659119 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.661902 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659122 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659124 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659127 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659129 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659131 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659134 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659136 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659139 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659141 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659144 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659146 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659150 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659153 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659155 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659158 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659161 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659164 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659166 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659171 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.662408 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659174 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659178 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659180 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659183 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659186 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659189 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659192 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659194 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659197 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659199 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659202 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659204 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659207 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659210 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659212 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659215 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659217 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659220 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659222 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659225 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.662869 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659227 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659230 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659232 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659236 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659239 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659243 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659246 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659249 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659255 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659258 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659261 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659264 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659268 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659271 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659274 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659277 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659280 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659283 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659286 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659288 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.663394 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659291 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.659293 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659392 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659412 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659421 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659426 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659430 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659433 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659438 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659442 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659445 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659448 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659451 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659455 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659458 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659461 2575 flags.go:64] FLAG: --cgroup-root="" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659464 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659467 2575 flags.go:64] FLAG: --client-ca-file="" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659470 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659474 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659480 2575 flags.go:64] FLAG: --cluster-domain="" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659483 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659487 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659490 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 13:16:41.663872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659493 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659498 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659500 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659504 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659507 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659510 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659513 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659516 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659519 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659522 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659527 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659530 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659533 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659535 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659538 2575 flags.go:64] FLAG: --enable-server="true" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659541 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659546 2575 flags.go:64] FLAG: --event-burst="100" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659549 2575 flags.go:64] FLAG: --event-qps="50" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659552 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659555 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659558 2575 flags.go:64] FLAG: --eviction-hard="" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659562 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659565 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659568 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659571 2575 flags.go:64] FLAG: --eviction-soft="" Jul 20 13:16:41.664450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659574 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659576 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659582 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659586 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659590 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659592 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659596 2575 flags.go:64] FLAG: --feature-gates="" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659599 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659602 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659606 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659609 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659611 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659615 2575 flags.go:64] FLAG: --help="false" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659617 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-142-97.ec2.internal" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659620 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659623 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659626 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659630 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659633 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659636 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659639 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659642 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659645 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 13:16:41.665037 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659647 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659651 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659653 2575 flags.go:64] FLAG: --kube-reserved="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659656 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659659 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659662 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659665 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659668 2575 flags.go:64] FLAG: --lock-file="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659671 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659674 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659677 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659682 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659689 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659692 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659695 2575 flags.go:64] FLAG: --logging-format="text" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659698 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659701 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659704 2575 flags.go:64] FLAG: --manifest-url="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659707 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659711 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659714 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659718 2575 flags.go:64] FLAG: --max-pods="110" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659721 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659724 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659726 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 13:16:41.665635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659730 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659733 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659736 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659739 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659746 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659749 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659752 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659755 2575 flags.go:64] FLAG: --pod-cidr="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659758 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659764 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659767 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659773 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659775 2575 flags.go:64] FLAG: --port="10250" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659778 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659781 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07a251725f3b34436" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659785 2575 flags.go:64] FLAG: --qos-reserved="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659787 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659790 2575 flags.go:64] FLAG: --register-node="true" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659793 2575 flags.go:64] FLAG: --register-with-taints="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659798 2575 flags.go:64] FLAG: --registry-burst="10" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659801 2575 flags.go:64] FLAG: --registry-qps="5" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659804 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659807 2575 flags.go:64] FLAG: --reserved-memory="" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659811 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659814 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 13:16:41.666258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659817 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659820 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659823 2575 flags.go:64] FLAG: --runonce="false" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659826 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659829 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659832 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659835 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659837 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659840 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659843 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659846 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659849 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659852 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659854 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659857 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659860 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659863 2575 flags.go:64] FLAG: --system-cgroups="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659866 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659873 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659876 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659879 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659883 2575 flags.go:64] FLAG: --tls-min-version="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659886 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659889 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659892 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 13:16:41.666921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659895 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659899 2575 flags.go:64] FLAG: --v="2" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659906 2575 flags.go:64] FLAG: --version="false" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659911 2575 flags.go:64] FLAG: --vmodule="" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659915 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.659918 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660037 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660041 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660044 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660047 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660050 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660053 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660056 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660058 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660061 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660063 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660066 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660069 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660071 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660074 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660077 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.667545 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660079 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660082 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660085 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660087 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660092 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660094 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660097 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660100 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660102 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660105 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660107 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660110 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660113 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660116 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660119 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660122 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660125 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660127 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660131 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.668067 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660135 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660138 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660141 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660143 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660146 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660148 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660152 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660155 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660158 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660162 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660166 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660168 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660171 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660174 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660176 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660179 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660181 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660187 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660190 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.668543 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660192 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660195 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660198 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660200 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660203 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660205 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660208 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660211 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660213 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660216 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660219 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660222 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660225 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660227 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660230 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660232 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660235 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660238 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660240 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660243 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.669009 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660246 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660249 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660251 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660254 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660257 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660259 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660262 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660264 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660266 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660269 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660273 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660275 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660278 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660281 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660283 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660286 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660288 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660291 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660294 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660297 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.669506 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660299 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660302 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660304 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660308 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660311 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.660313 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.660951 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.668165 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.668183 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.668964 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.668972 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669036 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669042 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669045 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.669991 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669050 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669055 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669058 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669062 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669065 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669068 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669071 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669074 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669077 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669080 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669083 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669085 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669088 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669091 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669093 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669096 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669098 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669101 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669104 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669106 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.670410 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669109 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669111 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669114 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669117 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669120 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669123 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669127 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669129 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669132 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669135 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669137 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669140 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669142 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669145 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669147 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669150 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669153 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669155 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669158 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669160 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.670897 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669163 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669165 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669169 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669172 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669174 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669177 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669179 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669182 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669185 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669187 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669190 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669192 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669195 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669197 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669200 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669202 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669206 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669210 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669214 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.671395 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669217 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669220 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669222 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669225 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669227 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669230 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669232 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669235 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669238 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669240 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669243 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669245 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669248 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669250 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669253 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669255 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669258 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669261 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669263 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.671857 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669266 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669270 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669274 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669277 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669280 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669282 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669285 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669287 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669290 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669293 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669295 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669298 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669301 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669303 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669306 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669310 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669312 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.672323 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669315 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.669320 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669454 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669461 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669465 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669468 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669471 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669474 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669477 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669480 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669483 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669486 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669489 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669492 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669494 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:41.672783 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669497 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669500 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669502 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669505 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669507 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669510 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669513 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669515 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669518 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669521 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669524 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669527 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669529 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669532 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669535 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669538 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669541 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669544 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669546 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:41.673148 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669549 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669552 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669555 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669557 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669560 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669563 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669565 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669568 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669570 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669573 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669576 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669578 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669581 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669583 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669586 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669589 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669591 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669594 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669597 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669599 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:41.673700 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669602 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669604 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669607 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669609 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669612 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669614 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669617 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669619 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669622 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669625 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669628 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669631 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669633 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669637 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669640 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669643 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669646 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669649 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669651 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:41.674243 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669654 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669656 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669659 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669662 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669664 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669667 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669669 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669672 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669674 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669677 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669680 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669683 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669685 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669689 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669691 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669694 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669696 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669699 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669702 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669704 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:41.674716 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669707 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669709 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669712 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669715 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669717 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669721 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669725 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:41.669728 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.669733 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.669892 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.671963 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.673395 2575 server.go:1019] "Starting client certificate rotation" Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.673494 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:41.675199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.673524 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:41.676478 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.676463 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:41.678004 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.677990 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:41.694570 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.694540 2575 log.go:25] "Validated CRI v1 runtime API" Jul 20 13:16:41.696832 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.696814 2575 log.go:25] "Validated CRI v1 image API" Jul 20 13:16:41.697597 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.697582 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 13:16:41.700915 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.700897 2575 fs.go:135] Filesystem UUIDs: map[5aa5f10b-b461-4864-9a7a-7f6dfa08344d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a2d04010-48b3-43d9-b95f-a8e366d90cc7:/dev/nvme0n1p3] Jul 20 13:16:41.700976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.700916 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 13:16:41.707429 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.707304 2575 manager.go:216] Machine: {Timestamp:2026-07-20 13:16:41.706029921 +0000 UTC m=+0.371968947 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099713 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25c554401b9d412da5df8b9d033688 SystemUUID:ec25c554-401b-9d41-2da5-df8b9d033688 BootID:fda7c155-142c-4fd1-a6b3-e32626487544 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:62:a0:81:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:62:a0:81:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:7c:5a:fc:29:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 13:16:41.707429 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.707421 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 13:16:41.707606 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.707501 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 13:16:41.708545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.708520 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 13:16:41.708694 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.708547 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 13:16:41.709404 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.709393 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 13:16:41.709445 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.709409 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 13:16:41.709445 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.709417 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:41.709445 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.709431 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:41.711090 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.711078 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 13:16:41.712647 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.712627 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:41.714160 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714141 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:41.714478 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714466 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 13:16:41.714526 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714499 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 20 13:16:41.714526 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714510 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 13:16:41.714599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714530 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 13:16:41.714599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714539 2575 kubelet.go:398] "Adding apiserver pod source" Jul 20 13:16:41.714599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714555 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 13:16:41.714922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714909 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:41.714957 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.714928 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:41.719068 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.719048 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 13:16:41.720478 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.720465 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 13:16:41.720530 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.720495 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 13:16:41.722496 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722482 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 13:16:41.722550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722505 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 13:16:41.722550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722520 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 13:16:41.722550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722529 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 13:16:41.722550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722540 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 13:16:41.722550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722545 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722554 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722560 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722567 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722572 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722582 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 13:16:41.722740 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.722588 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 13:16:41.723473 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.723453 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 13:16:41.723473 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.723468 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 13:16:41.724531 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.724510 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 13:16:41.725574 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.725542 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 13:16:41.726869 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.726855 2575 server.go:1268] "Started kubelet" Jul 20 13:16:41.727005 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.726980 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 13:16:41.727546 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.727487 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 13:16:41.727641 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.727574 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 13:16:41.727975 ip-10-0-142-97 systemd[1]: Started Kubernetes Kubelet. Jul 20 13:16:41.729383 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.729345 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 13:16:41.731284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.731258 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 20 13:16:41.733012 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.732991 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 13:16:41.735961 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.735940 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 13:16:41.737760 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.736004 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-97.ec2.internal.18c401dd28a47830 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-97.ec2.internal,UID:ip-10-0-142-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-97.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.726826544 +0000 UTC m=+0.392765566,LastTimestamp:2026-07-20 13:16:41.726826544 +0000 UTC m=+0.392765566,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-97.ec2.internal,}" Jul 20 13:16:41.740452 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.740437 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 13:16:41.740452 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.740444 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:41.740557 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.740520 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:41.741374 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741324 2575 factory.go:55] Registering systemd factory Jul 20 13:16:41.741374 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741344 2575 factory.go:223] Registration of the systemd container factory successfully Jul 20 13:16:41.741524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741435 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 13:16:41.741524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741478 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 13:16:41.741524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741494 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 13:16:41.741659 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741635 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 20 13:16:41.741659 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.741644 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 20 13:16:41.742304 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742288 2575 factory.go:153] Registering CRI-O factory Jul 20 13:16:41.742304 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742305 2575 factory.go:223] Registration of the crio container factory successfully Jul 20 13:16:41.742464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742388 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 13:16:41.742464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742415 2575 factory.go:103] Registering Raw factory Jul 20 13:16:41.742464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742435 2575 manager.go:1196] Started watching for new ooms in manager Jul 20 13:16:41.742883 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.742832 2575 manager.go:349] Starting recovery of all containers Jul 20 13:16:41.744325 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.744299 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 13:16:41.744731 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.744693 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 13:16:41.753849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.753705 2575 manager.go:354] Recovery completed Jul 20 13:16:41.758599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.758586 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.760583 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.760555 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.760697 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.760597 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.760697 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.760611 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.761145 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761128 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 13:16:41.761145 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761141 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 13:16:41.761267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761151 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.761267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761159 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:41.761920 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761904 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.761991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761930 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.761991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.761943 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.762475 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.762464 2575 policy_none.go:49] "None policy: Start" Jul 20 13:16:41.762523 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.762484 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 13:16:41.762523 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.762495 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 13:16:41.762905 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.762808 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-97.ec2.internal.18c401dd2aa7841d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-97.ec2.internal,UID:ip-10-0-142-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-97.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.760580637 +0000 UTC m=+0.426519658,LastTimestamp:2026-07-20 13:16:41.760580637 +0000 UTC m=+0.426519658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-97.ec2.internal,}" Jul 20 13:16:41.763742 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.763731 2575 policy_none.go:47] "Start" Jul 20 13:16:41.768672 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.768648 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7njhq" Jul 20 13:16:41.774985 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.774911 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-97.ec2.internal.18c401dd2aa7e4f8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-97.ec2.internal,UID:ip-10-0-142-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-97.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-97.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.760605432 +0000 UTC m=+0.426544454,LastTimestamp:2026-07-20 13:16:41.760605432 +0000 UTC m=+0.426544454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-97.ec2.internal,}" Jul 20 13:16:41.781543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.781521 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7njhq" Jul 20 13:16:41.787545 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.787477 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-97.ec2.internal.18c401dd2aa80b74 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-97.ec2.internal,UID:ip-10-0-142-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-97.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-97.ec2.internal,},FirstTimestamp:2026-07-20 13:16:41.760615284 +0000 UTC m=+0.426554309,LastTimestamp:2026-07-20 13:16:41.760615284 +0000 UTC m=+0.426554309,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-97.ec2.internal,}" Jul 20 13:16:41.802024 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802001 2575 manager.go:337] "Starting Device Plugin manager" Jul 20 13:16:41.802128 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.802043 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 13:16:41.802128 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802053 2575 server.go:87] "Starting device plugin registration server" Jul 20 13:16:41.802399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802386 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 13:16:41.802437 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802402 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 13:16:41.802519 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802502 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 13:16:41.802619 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802600 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 13:16:41.802619 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.802615 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 13:16:41.803208 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.803185 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 13:16:41.803289 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.803237 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:41.874030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.873998 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 13:16:41.875174 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.875158 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 13:16:41.875261 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.875210 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 13:16:41.875261 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.875239 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 13:16:41.875370 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.875282 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 13:16:41.877784 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.877760 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:41.902998 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.902919 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.904113 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.904094 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.904293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.904126 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.904293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.904140 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.904293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.904164 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-97.ec2.internal" Jul 20 13:16:41.914328 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.914300 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-97.ec2.internal" Jul 20 13:16:41.914328 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.914325 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-97.ec2.internal\": node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:41.943786 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.943759 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:41.975612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.975583 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal"] Jul 20 13:16:41.975694 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.975657 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.977286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.977269 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.977365 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.977300 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.977365 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.977310 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.978649 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.978636 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.978792 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.978778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:41.978836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.978808 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.979392 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979375 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.979489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979408 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.979489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979419 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.979489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979379 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.979489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979472 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.979489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.979481 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.980596 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.980582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:41.980657 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.980606 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:41.981420 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.981403 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:41.981518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.981434 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:41.981518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:41.981446 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:41.998013 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:41.997990 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-97.ec2.internal\" not found" node="ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.002726 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.002708 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-97.ec2.internal\" not found" node="ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.043433 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.043395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.043433 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.043432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.043636 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.043450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3259959531b2cae4a95970f39ca273a7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-97.ec2.internal\" (UID: \"3259959531b2cae4a95970f39ca273a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.044422 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.044405 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.144202 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.144202 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.144461 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3259959531b2cae4a95970f39ca273a7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-97.ec2.internal\" (UID: \"3259959531b2cae4a95970f39ca273a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.144461 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.144461 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3259959531b2cae4a95970f39ca273a7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-97.ec2.internal\" (UID: \"3259959531b2cae4a95970f39ca273a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.144461 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.144320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3fe2c8d6a8514d87ee3bfa611989887-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal\" (UID: \"c3fe2c8d6a8514d87ee3bfa611989887\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.145201 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.145184 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.246006 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.245934 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.301153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.301122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.305737 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.305713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.346285 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.346252 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.446743 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.446702 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.547290 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.547223 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-97.ec2.internal\" not found" Jul 20 13:16:42.561224 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.561195 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:42.616933 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.616901 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:42.641213 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.641183 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.653280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.653261 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:42.654222 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.654207 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" Jul 20 13:16:42.662872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.662853 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:42.673012 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.672992 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 13:16:42.673126 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.673108 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 13:16:42.673173 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.673147 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 13:16:42.673173 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.673157 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 13:16:42.715286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.715254 2575 apiserver.go:52] "Watching apiserver" Jul 20 13:16:42.734427 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.734391 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 13:16:42.734753 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.734730 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4pdbn","openshift-multus/network-metrics-daemon-6kb7r","openshift-network-diagnostics/network-check-target-knxpg","kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl","openshift-cluster-node-tuning-operator/tuned-z2hsz","openshift-image-registry/node-ca-stgxm","openshift-multus/multus-additional-cni-plugins-ddzwq","openshift-network-operator/iptables-alerter-tsbss","openshift-ovn-kubernetes/ovnkube-node-w2nmh","kube-system/global-pull-secret-syncer-rvdwx","kube-system/konnectivity-agent-8dgkv","openshift-dns/node-resolver-drxgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal"] Jul 20 13:16:42.736157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.736138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.737627 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.737609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.738937 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.738918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 13:16:42.739042 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.738952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.739042 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.738981 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 13:16:42.739042 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.739007 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.739196 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.739041 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjmd5\"" Jul 20 13:16:42.739196 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.739082 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 13:16:42.739196 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.738982 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.740143 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.740127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.740252 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.740174 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.740316 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.740286 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfznd\"" Jul 20 13:16:42.740403 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.740389 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.740518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.740504 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:42.741322 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.741305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.741732 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.741712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8bpdd\"" Jul 20 13:16:42.741848 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.741717 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 13:16:42.741848 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.741786 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 13:16:42.742472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.742451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.742552 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.742535 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 13:16:42.742689 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.742668 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kskvr\"" Jul 20 13:16:42.743610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.743595 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sq28r\"" Jul 20 13:16:42.743693 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.743595 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.743757 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.743745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.744087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.744713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744698 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 13:16:42.744864 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744849 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.744981 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.744981 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744978 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlwwv\"" Jul 20 13:16:42.745085 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.744994 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.746123 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746329 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-host\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbq9p\" (UniqueName: \"kubernetes.io/projected/2f4c931e-a517-44a1-a18a-928eb98dcdf6-kube-api-access-fbq9p\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-multus\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-etc-kubernetes\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.746563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fc35a62-f9fe-46d9-8b66-3d5115c12503-iptables-alerter-script\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-systemd\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-sys\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-lib-modules\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cnibin\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-os-release\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d83c7a14-b7c1-4951-a080-f57d421fd337-agent-certs\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.746856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-modprobe-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-run\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-tmp\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9cb3873-9970-4689-9e1f-423a90921969-hosts-file\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746932 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-daemon-config\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746959 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tznzm\"" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvnlz\" (UniqueName: \"kubernetes.io/projected/8fc35a62-f9fe-46d9-8b66-3d5115c12503-kube-api-access-dvnlz\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.746999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysconfig\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5sq\" (UniqueName: \"kubernetes.io/projected/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-kube-api-access-cf5sq\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4tq\" (UniqueName: \"kubernetes.io/projected/f9cb3873-9970-4689-9e1f-423a90921969-kube-api-access-zp4tq\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-cni-binary-copy\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-netns\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747153 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 13:16:42.747187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-tuned\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-k8s-cni-cncf-io\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-hostroot\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-conf-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-multus-certs\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-system-cni-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cb3873-9970-4689-9e1f-423a90921969-tmp-dir\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d83c7a14-b7c1-4951-a080-f57d421fd337-konnectivity-ca\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-system-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747564 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8h8gr\"" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-cnibin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-bin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.747647 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-kubelet\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.747781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747654 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747677 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fc35a62-f9fe-46d9-8b66-3d5115c12503-host-slash\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-os-release\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-socket-dir-parent\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwtj2\" (UniqueName: \"kubernetes.io/projected/9224d74e-7bc8-4633-a751-43a52e050fcf-kube-api-access-bwtj2\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-kubernetes\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-conf\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.748343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.747905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-var-lib-kubelet\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.748985 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.748951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:42.749052 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.749026 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 13:16:42.749052 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.749032 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:42.750090 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.750076 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 13:16:42.750301 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.750287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:42.750425 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.750342 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:42.751377 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.751049 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-slbzm\"" Jul 20 13:16:42.751377 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.751058 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 13:16:42.751798 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.751699 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 13:16:42.751798 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.751734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 13:16:42.751798 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.751790 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 13:16:42.756342 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.756319 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:42.777598 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.777577 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-smmmq" Jul 20 13:16:42.786528 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.786470 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 13:11:41 +0000 UTC" deadline="2028-01-08 00:11:49.930345343 +0000 UTC" Jul 20 13:16:42.786528 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.786525 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12874h55m7.143822926s" Jul 20 13:16:42.786698 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.786661 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-smmmq" Jul 20 13:16:42.842515 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.842494 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 13:16:42.848797 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-netns\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-hostroot\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-conf-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-systemd-units\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-hostroot\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-conf-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.848925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-device-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-netns\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.848961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-k8s-cni-cncf-io\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-k8s-cni-cncf-io\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-multus-certs\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-kubelet-config\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-run-multus-certs\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-system-cni-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-system-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-kubelet\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-kubelet\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-system-cni-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-env-overrides\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-kubelet\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-system-cni-dir\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld49\" (UniqueName: \"kubernetes.io/projected/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kube-api-access-kld49\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fc35a62-f9fe-46d9-8b66-3d5115c12503-host-slash\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8fc35a62-f9fe-46d9-8b66-3d5115c12503-host-slash\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.849524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwtj2\" (UniqueName: \"kubernetes.io/projected/9224d74e-7bc8-4633-a751-43a52e050fcf-kube-api-access-bwtj2\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80bdd741-0dab-4291-9470-e3a679e6b988-ovn-node-metrics-cert\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-netns\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-conf\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-host\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-multus\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-etc-kubernetes\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-node-log\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmj4d\" (UniqueName: \"kubernetes.io/projected/80bdd741-0dab-4291-9470-e3a679e6b988-kube-api-access-bmj4d\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-multus\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-lib-modules\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysctl-conf\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.849984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.850239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-etc-kubernetes\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-host\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d83c7a14-b7c1-4951-a080-f57d421fd337-agent-certs\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850060 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-lib-modules\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-daemon-config\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850163 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-slash\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-log-socket\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-config\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97144075-89d8-4105-a155-f224c2724c68-host\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-run\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9cb3873-9970-4689-9e1f-423a90921969-hosts-file\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrvsw\" (UniqueName: \"kubernetes.io/projected/97144075-89d8-4105-a155-f224c2724c68-kube-api-access-mrvsw\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850410 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-run\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-sys-fs\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9cb3873-9970-4689-9e1f-423a90921969-hosts-file\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850445 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 13:16:42.850921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-dbus\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxx9\" (UniqueName: \"kubernetes.io/projected/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-kube-api-access-jcxx9\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvnlz\" (UniqueName: \"kubernetes.io/projected/8fc35a62-f9fe-46d9-8b66-3d5115c12503-kube-api-access-dvnlz\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850650 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-daemon-config\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5sq\" (UniqueName: \"kubernetes.io/projected/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-kube-api-access-cf5sq\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-tuned\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cb3873-9970-4689-9e1f-423a90921969-tmp-dir\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d83c7a14-b7c1-4951-a080-f57d421fd337-konnectivity-ca\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-cnibin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-bin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-var-lib-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-host-var-lib-cni-bin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-script-lib\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-cnibin\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.850968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-socket-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.851748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-os-release\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-socket-dir-parent\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-os-release\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-etc-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-ovn\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9224d74e-7bc8-4633-a751-43a52e050fcf-multus-socket-dir-parent\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-kubernetes\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-var-lib-kubelet\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f4c931e-a517-44a1-a18a-928eb98dcdf6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbq9p\" (UniqueName: \"kubernetes.io/projected/2f4c931e-a517-44a1-a18a-928eb98dcdf6-kube-api-access-fbq9p\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-kubernetes\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97144075-89d8-4105-a155-f224c2724c68-serviceca\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-var-lib-kubelet\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fc35a62-f9fe-46d9-8b66-3d5115c12503-iptables-alerter-script\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cb3873-9970-4689-9e1f-423a90921969-tmp-dir\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-systemd\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-systemd\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.852550 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-sys\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cnibin\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-os-release\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-sys\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-systemd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-bin\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-cnibin\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-modprobe-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-tmp\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-netd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-registration-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851638 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-modprobe-d\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysconfig\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4tq\" (UniqueName: \"kubernetes.io/projected/f9cb3873-9970-4689-9e1f-423a90921969-kube-api-access-zp4tq\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2f4c931e-a517-44a1-a18a-928eb98dcdf6-os-release\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-cni-binary-copy\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.853147 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8fc35a62-f9fe-46d9-8b66-3d5115c12503-iptables-alerter-script\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.853821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.851931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-sysconfig\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.852140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d83c7a14-b7c1-4951-a080-f57d421fd337-konnectivity-ca\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.853821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.852229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9224d74e-7bc8-4633-a751-43a52e050fcf-cni-binary-copy\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.853821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.853788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-tmp\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.853821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.853817 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-etc-tuned\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.854056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.853987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d83c7a14-b7c1-4951-a080-f57d421fd337-agent-certs\") pod \"konnectivity-agent-8dgkv\" (UID: \"d83c7a14-b7c1-4951-a080-f57d421fd337\") " pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:42.859121 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.859101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5sq\" (UniqueName: \"kubernetes.io/projected/6e6c8066-46aa-4049-a5ba-97a303a1b5e4-kube-api-access-cf5sq\") pod \"tuned-z2hsz\" (UID: \"6e6c8066-46aa-4049-a5ba-97a303a1b5e4\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:42.859258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.859151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwtj2\" (UniqueName: \"kubernetes.io/projected/9224d74e-7bc8-4633-a751-43a52e050fcf-kube-api-access-bwtj2\") pod \"multus-4pdbn\" (UID: \"9224d74e-7bc8-4633-a751-43a52e050fcf\") " pod="openshift-multus/multus-4pdbn" Jul 20 13:16:42.859815 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.859791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvnlz\" (UniqueName: \"kubernetes.io/projected/8fc35a62-f9fe-46d9-8b66-3d5115c12503-kube-api-access-dvnlz\") pod \"iptables-alerter-tsbss\" (UID: \"8fc35a62-f9fe-46d9-8b66-3d5115c12503\") " pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:42.860148 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.860127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4tq\" (UniqueName: \"kubernetes.io/projected/f9cb3873-9970-4689-9e1f-423a90921969-kube-api-access-zp4tq\") pod \"node-resolver-drxgx\" (UID: \"f9cb3873-9970-4689-9e1f-423a90921969\") " pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:42.860337 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.860322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbq9p\" (UniqueName: \"kubernetes.io/projected/2f4c931e-a517-44a1-a18a-928eb98dcdf6-kube-api-access-fbq9p\") pod \"multus-additional-cni-plugins-ddzwq\" (UID: \"2f4c931e-a517-44a1-a18a-928eb98dcdf6\") " pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:42.865878 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.865856 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:42.886259 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:42.886227 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fe2c8d6a8514d87ee3bfa611989887.slice/crio-5c69eaef7a05287a978e4cefde9f53bbfabd8ba054d18b3edf6194380e0b2f37 WatchSource:0}: Error finding container 5c69eaef7a05287a978e4cefde9f53bbfabd8ba054d18b3edf6194380e0b2f37: Status 404 returned error can't find the container with id 5c69eaef7a05287a978e4cefde9f53bbfabd8ba054d18b3edf6194380e0b2f37 Jul 20 13:16:42.886559 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:42.886538 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3259959531b2cae4a95970f39ca273a7.slice/crio-2896e53458a7db51fad90e69fcd05f2033d0b470902a1328ae4187494f3ceecc WatchSource:0}: Error finding container 2896e53458a7db51fad90e69fcd05f2033d0b470902a1328ae4187494f3ceecc: Status 404 returned error can't find the container with id 2896e53458a7db51fad90e69fcd05f2033d0b470902a1328ae4187494f3ceecc Jul 20 13:16:42.891941 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.891918 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:16:42.952204 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-etc-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952204 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-ovn\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952204 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97144075-89d8-4105-a155-f224c2724c68-serviceca\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-systemd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-bin\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-bin\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-etc-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-ovn\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-netd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-cni-netd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-registration-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-systemd\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-systemd-units\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952455 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-registration-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-device-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-systemd-units\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-kubelet-config\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-device-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.952559 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-kubelet\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-kubelet-config\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-env-overrides\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.952554 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld49\" (UniqueName: \"kubernetes.io/projected/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kube-api-access-kld49\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.952637 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.452612993 +0000 UTC m=+2.118552023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-kubelet\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97144075-89d8-4105-a155-f224c2724c68-serviceca\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-run-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.953335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80bdd741-0dab-4291-9470-e3a679e6b988-ovn-node-metrics-cert\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-netns\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-node-log\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmj4d\" (UniqueName: \"kubernetes.io/projected/80bdd741-0dab-4291-9470-e3a679e6b988-kube-api-access-bmj4d\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-slash\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-log-socket\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-env-overrides\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.952978 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.952978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-config\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-run-netns\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97144075-89d8-4105-a155-f224c2724c68-host\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-slash\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.953045 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.453029582 +0000 UTC m=+2.118968601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-log-socket\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97144075-89d8-4105-a155-f224c2724c68-host\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrvsw\" (UniqueName: \"kubernetes.io/projected/97144075-89d8-4105-a155-f224c2724c68-kube-api-access-mrvsw\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.954020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-node-log\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-sys-fs\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-dbus\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-sys-fs\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxx9\" (UniqueName: \"kubernetes.io/projected/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-kube-api-access-jcxx9\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-var-lib-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-script-lib\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e8945d1-09f8-44da-aeca-94ff385f58f8-dbus\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-socket-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80bdd741-0dab-4291-9470-e3a679e6b988-var-lib-openvswitch\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-socket-dir\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953586 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-config\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.954548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.953714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80bdd741-0dab-4291-9470-e3a679e6b988-ovnkube-script-lib\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.955256 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.955240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80bdd741-0dab-4291-9470-e3a679e6b988-ovn-node-metrics-cert\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.964821 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.964805 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:42.964918 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.964825 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:42.964918 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.964835 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:42.964918 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:42.964887 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:16:43.46487531 +0000 UTC m=+2.130814333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:42.966265 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.966241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld49\" (UniqueName: \"kubernetes.io/projected/ea5539e4-8b05-40ee-b1c3-da73d8c3502f-kube-api-access-kld49\") pod \"aws-ebs-csi-driver-node-fdqbl\" (UID: \"ea5539e4-8b05-40ee-b1c3-da73d8c3502f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:42.967260 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.967243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmj4d\" (UniqueName: \"kubernetes.io/projected/80bdd741-0dab-4291-9470-e3a679e6b988-kube-api-access-bmj4d\") pod \"ovnkube-node-w2nmh\" (UID: \"80bdd741-0dab-4291-9470-e3a679e6b988\") " pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:42.969847 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.969827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrvsw\" (UniqueName: \"kubernetes.io/projected/97144075-89d8-4105-a155-f224c2724c68-kube-api-access-mrvsw\") pod \"node-ca-stgxm\" (UID: \"97144075-89d8-4105-a155-f224c2724c68\") " pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:42.969922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:42.969887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxx9\" (UniqueName: \"kubernetes.io/projected/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-kube-api-access-jcxx9\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:43.077916 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.077798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" Jul 20 13:16:43.086173 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.086141 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4c931e_a517_44a1_a18a_928eb98dcdf6.slice/crio-d3f1aa768cb08fa72d998e443329b97cac872a1a53b8fbdcb67e162f2e5d18d9 WatchSource:0}: Error finding container d3f1aa768cb08fa72d998e443329b97cac872a1a53b8fbdcb67e162f2e5d18d9: Status 404 returned error can't find the container with id d3f1aa768cb08fa72d998e443329b97cac872a1a53b8fbdcb67e162f2e5d18d9 Jul 20 13:16:43.099106 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.099086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drxgx" Jul 20 13:16:43.106161 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.106132 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cb3873_9970_4689_9e1f_423a90921969.slice/crio-cdfcb156a2f4f8ed89ad0303fb0e0c6e502bda46b15db748eb25bbdcf5a04709 WatchSource:0}: Error finding container cdfcb156a2f4f8ed89ad0303fb0e0c6e502bda46b15db748eb25bbdcf5a04709: Status 404 returned error can't find the container with id cdfcb156a2f4f8ed89ad0303fb0e0c6e502bda46b15db748eb25bbdcf5a04709 Jul 20 13:16:43.111161 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.111143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:16:43.115218 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.115199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pdbn" Jul 20 13:16:43.121452 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.121417 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83c7a14_b7c1_4951_a080_f57d421fd337.slice/crio-53415939434fe71fa0c75bfa35f5580ef53b71e9a35fee2c7c4a0482377dae30 WatchSource:0}: Error finding container 53415939434fe71fa0c75bfa35f5580ef53b71e9a35fee2c7c4a0482377dae30: Status 404 returned error can't find the container with id 53415939434fe71fa0c75bfa35f5580ef53b71e9a35fee2c7c4a0482377dae30 Jul 20 13:16:43.124531 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.124511 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224d74e_7bc8_4633_a751_43a52e050fcf.slice/crio-0bdb5d0753b58b239265827fc0ae41978516ffb312523e658faeb90880a2f3fd WatchSource:0}: Error finding container 0bdb5d0753b58b239265827fc0ae41978516ffb312523e658faeb90880a2f3fd: Status 404 returned error can't find the container with id 0bdb5d0753b58b239265827fc0ae41978516ffb312523e658faeb90880a2f3fd Jul 20 13:16:43.129433 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.129412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tsbss" Jul 20 13:16:43.136818 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.136797 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc35a62_f9fe_46d9_8b66_3d5115c12503.slice/crio-da6e585e6232e1eeee9232104874afe3fe7f967c4816dd4a12bf7836db76787d WatchSource:0}: Error finding container da6e585e6232e1eeee9232104874afe3fe7f967c4816dd4a12bf7836db76787d: Status 404 returned error can't find the container with id da6e585e6232e1eeee9232104874afe3fe7f967c4816dd4a12bf7836db76787d Jul 20 13:16:43.142907 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.142887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" Jul 20 13:16:43.149546 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.149526 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6c8066_46aa_4049_a5ba_97a303a1b5e4.slice/crio-d6e24440ab6ecadad81abb08325cc4445075003c2f552f60c21677f7ce300589 WatchSource:0}: Error finding container d6e24440ab6ecadad81abb08325cc4445075003c2f552f60c21677f7ce300589: Status 404 returned error can't find the container with id d6e24440ab6ecadad81abb08325cc4445075003c2f552f60c21677f7ce300589 Jul 20 13:16:43.167695 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.167670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stgxm" Jul 20 13:16:43.175253 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.175228 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97144075_89d8_4105_a155_f224c2724c68.slice/crio-a28b3bb823b7533ba908d5ec500e8e053ecef98c47533ccec4cce08938499e50 WatchSource:0}: Error finding container a28b3bb823b7533ba908d5ec500e8e053ecef98c47533ccec4cce08938499e50: Status 404 returned error can't find the container with id a28b3bb823b7533ba908d5ec500e8e053ecef98c47533ccec4cce08938499e50 Jul 20 13:16:43.185645 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.185626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" Jul 20 13:16:43.190950 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.190932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:16:43.193318 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.193295 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5539e4_8b05_40ee_b1c3_da73d8c3502f.slice/crio-afdf30f78d6441aad6b453e910c1fe3df58020de89b9927432bc51b0161164e6 WatchSource:0}: Error finding container afdf30f78d6441aad6b453e910c1fe3df58020de89b9927432bc51b0161164e6: Status 404 returned error can't find the container with id afdf30f78d6441aad6b453e910c1fe3df58020de89b9927432bc51b0161164e6 Jul 20 13:16:43.198189 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:16:43.198170 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bdd741_0dab_4291_9470_e3a679e6b988.slice/crio-56f43f6b3ff1771f0556c3705867a7c84341f65108946233f0b38c7cc2172f55 WatchSource:0}: Error finding container 56f43f6b3ff1771f0556c3705867a7c84341f65108946233f0b38c7cc2172f55: Status 404 returned error can't find the container with id 56f43f6b3ff1771f0556c3705867a7c84341f65108946233f0b38c7cc2172f55 Jul 20 13:16:43.456703 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.456612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:43.456898 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.456767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:43.456898 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.456770 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.456898 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.456841 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:44.456823108 +0000 UTC m=+3.122762137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:43.456898 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.456893 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.457115 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.456954 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:44.456938932 +0000 UTC m=+3.122877954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:43.543603 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.543562 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:43.557326 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.557289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:43.557561 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.557453 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:43.557561 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.557472 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:43.557561 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.557480 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:43.557561 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.557532 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:16:44.557516743 +0000 UTC m=+3.223455774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:43.789084 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.788996 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:42 +0000 UTC" deadline="2028-02-08 20:43:27.646295002 +0000 UTC" Jul 20 13:16:43.789084 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.789034 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13639h26m43.857265345s" Jul 20 13:16:43.878183 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.876015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:43.878183 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.876045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:43.878183 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.876147 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:43.878183 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:43.876297 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:43.892348 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.892289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stgxm" event={"ID":"97144075-89d8-4105-a155-f224c2724c68","Type":"ContainerStarted","Data":"a28b3bb823b7533ba908d5ec500e8e053ecef98c47533ccec4cce08938499e50"} Jul 20 13:16:43.911417 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.911376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" event={"ID":"6e6c8066-46aa-4049-a5ba-97a303a1b5e4","Type":"ContainerStarted","Data":"d6e24440ab6ecadad81abb08325cc4445075003c2f552f60c21677f7ce300589"} Jul 20 13:16:43.925247 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.925212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tsbss" event={"ID":"8fc35a62-f9fe-46d9-8b66-3d5115c12503","Type":"ContainerStarted","Data":"da6e585e6232e1eeee9232104874afe3fe7f967c4816dd4a12bf7836db76787d"} Jul 20 13:16:43.930042 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.930010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pdbn" event={"ID":"9224d74e-7bc8-4633-a751-43a52e050fcf","Type":"ContainerStarted","Data":"0bdb5d0753b58b239265827fc0ae41978516ffb312523e658faeb90880a2f3fd"} Jul 20 13:16:43.942020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.941987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8dgkv" event={"ID":"d83c7a14-b7c1-4951-a080-f57d421fd337","Type":"ContainerStarted","Data":"53415939434fe71fa0c75bfa35f5580ef53b71e9a35fee2c7c4a0482377dae30"} Jul 20 13:16:43.965154 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.964988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerStarted","Data":"d3f1aa768cb08fa72d998e443329b97cac872a1a53b8fbdcb67e162f2e5d18d9"} Jul 20 13:16:43.985759 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.985721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" event={"ID":"c3fe2c8d6a8514d87ee3bfa611989887","Type":"ContainerStarted","Data":"5c69eaef7a05287a978e4cefde9f53bbfabd8ba054d18b3edf6194380e0b2f37"} Jul 20 13:16:43.999477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:43.999430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"56f43f6b3ff1771f0556c3705867a7c84341f65108946233f0b38c7cc2172f55"} Jul 20 13:16:44.011131 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.011087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" event={"ID":"ea5539e4-8b05-40ee-b1c3-da73d8c3502f","Type":"ContainerStarted","Data":"afdf30f78d6441aad6b453e910c1fe3df58020de89b9927432bc51b0161164e6"} Jul 20 13:16:44.021711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.021544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drxgx" event={"ID":"f9cb3873-9970-4689-9e1f-423a90921969","Type":"ContainerStarted","Data":"cdfcb156a2f4f8ed89ad0303fb0e0c6e502bda46b15db748eb25bbdcf5a04709"} Jul 20 13:16:44.023703 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.023676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" event={"ID":"3259959531b2cae4a95970f39ca273a7","Type":"ContainerStarted","Data":"2896e53458a7db51fad90e69fcd05f2033d0b470902a1328ae4187494f3ceecc"} Jul 20 13:16:44.437824 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.436555 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.464675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.464745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.464872 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.464936 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:46.464915774 +0000 UTC m=+5.130854799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.465371 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:44.465472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.465418 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:46.465404193 +0000 UTC m=+5.131343208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:44.566086 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.566050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:44.566276 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.566234 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:44.566276 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.566256 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:44.566276 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.566268 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:44.566456 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.566325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:16:46.566306427 +0000 UTC m=+5.232245457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:44.605064 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.605032 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:44.789684 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.789572 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:42 +0000 UTC" deadline="2028-04-01 06:54:49.135943574 +0000 UTC" Jul 20 13:16:44.789684 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.789635 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14897h38m4.346312732s" Jul 20 13:16:44.875487 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:44.875452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:44.875664 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:44.875585 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:45.881228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:45.879038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:45.881228 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:45.879418 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:45.881228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:45.879870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:45.881228 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:45.879986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:46.486188 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:46.486154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:46.486400 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:46.486241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:46.486472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.486404 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:46.486472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.486469 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:50.486449046 +0000 UTC m=+9.152388078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:46.486870 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.486849 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:46.486947 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.486907 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:50.486892711 +0000 UTC m=+9.152831725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:46.587148 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:46.587082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:46.587326 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.587303 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:46.587326 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.587325 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:46.587453 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.587340 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:46.587453 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.587426 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:16:50.587406349 +0000 UTC m=+9.253345361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:46.876167 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:46.876090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:46.876325 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:46.876232 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:47.876048 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:47.875991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:47.876763 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:47.876167 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:47.876763 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:47.876396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:47.876763 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:47.876521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:48.876050 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:48.876008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:48.876539 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:48.876143 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:49.875931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:49.875706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:49.875931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:49.875732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:49.875931 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:49.875847 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:49.876537 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:49.876260 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:50.519746 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:50.519127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:50.519746 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:50.519194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:50.519746 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.519328 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:50.519746 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.519406 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:58.519388867 +0000 UTC m=+17.185327890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:50.519746 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.519732 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:50.520159 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.519798 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:58.519771383 +0000 UTC m=+17.185710411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:50.620331 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:50.620298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:50.620522 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.620468 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:50.620522 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.620485 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:50.620522 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.620494 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:50.620695 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.620554 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:16:58.620535742 +0000 UTC m=+17.286474774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:50.876430 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:50.876341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:50.876805 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:50.876489 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:51.877426 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:51.877387 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:51.877880 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:51.877509 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:51.877880 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:51.877867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:51.878011 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:51.877983 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:52.875730 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:52.875692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:52.875922 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:52.875800 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:53.876055 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:53.875970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:53.876509 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:53.876117 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:53.876509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:53.875970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:53.876509 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:53.876387 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:54.876396 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:54.876287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:54.876812 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:54.876448 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:55.875759 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:55.875718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:55.875957 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:55.875856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:55.875957 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:55.875907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:55.876067 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:55.876035 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:56.876265 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:56.876233 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:56.876688 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:56.876373 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:57.876259 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:57.876216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:57.876472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:57.876216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:57.876472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:57.876379 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:16:57.876472 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:57.876459 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:58.582702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:58.582665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:58.582912 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:58.582733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:58.582912 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.582836 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:58.583005 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.582913 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:14.582895194 +0000 UTC m=+33.248834214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:58.583005 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.582838 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:58.583005 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.582992 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:14.582974824 +0000 UTC m=+33.248913846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:58.683285 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:58.683251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:58.683473 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.683450 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:58.683541 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.683492 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:58.683541 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.683507 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:58.683645 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.683568 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:17:14.683553429 +0000 UTC m=+33.349492462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:58.876273 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:58.876197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:16:58.876452 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:58.876309 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:16:59.876138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:59.876098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:16:59.876708 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:16:59.876098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:16:59.876708 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:59.876246 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:16:59.876708 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:16:59.876330 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:00.875845 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:00.875812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:00.876023 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:00.875925 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:01.876699 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:01.876636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:01.877161 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:01.876764 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:01.877161 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:01.876847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:01.877161 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:01.876974 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:02.875849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:02.875822 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:02.875973 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:02.875953 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:03.075130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.074863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" event={"ID":"6e6c8066-46aa-4049-a5ba-97a303a1b5e4","Type":"ContainerStarted","Data":"4d6e218101fdd0e1f29352f7adca270990072416710c560f8cb520c6dd9f28fe"} Jul 20 13:17:03.083533 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.083462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pdbn" event={"ID":"9224d74e-7bc8-4633-a751-43a52e050fcf","Type":"ContainerStarted","Data":"473e1e0a4133b50e3f96f2c3a34ecd2dd1d94a976b17af10767572a91b03ce89"} Jul 20 13:17:03.086433 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.086333 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"89e87438333738c13a318dba60328479b604f92e75998027c5c55e2fda9efd5a"} Jul 20 13:17:03.088400 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.088349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" event={"ID":"3259959531b2cae4a95970f39ca273a7","Type":"ContainerStarted","Data":"a41a321c10a6127755c5b5118b05c0b2a784185f8041ec9e297fcab8865e3678"} Jul 20 13:17:03.095122 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.095051 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z2hsz" podStartSLOduration=2.395277423 podStartE2EDuration="22.095031519s" totalImagesPullingTime="19.699754096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.09466726 +0000 UTC m=+21.760606329" watchObservedRunningTime="2026-07-20 13:17:03.095031519 +0000 UTC m=+21.760970553" Jul 20 13:17:03.116634 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.116245 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4pdbn" podStartSLOduration=2.290598068 podStartE2EDuration="22.116225466s" totalImagesPullingTime="19.825627398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.115644213 +0000 UTC m=+21.781583244" watchObservedRunningTime="2026-07-20 13:17:03.116225466 +0000 UTC m=+21.782164497" Jul 20 13:17:03.158631 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.156382 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-97.ec2.internal" podStartSLOduration=21.156339458 podStartE2EDuration="21.156339458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:03.149022959 +0000 UTC m=+21.814961992" watchObservedRunningTime="2026-07-20 13:17:03.156339458 +0000 UTC m=+21.822278490" Jul 20 13:17:03.876624 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.876409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:03.876756 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:03.876431 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:03.876756 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:03.876670 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:03.876756 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:03.876732 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:04.090973 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.090931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8dgkv" event={"ID":"d83c7a14-b7c1-4951-a080-f57d421fd337","Type":"ContainerStarted","Data":"636f3f3e362728223059313480d4e81979e351a04854faac7125e37a7c3e533a"} Jul 20 13:17:04.092369 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.092336 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="4ff26fdc17148000d01b14184bd9197c37d30c6860610abdbdb86bb9b1e929b5" exitCode=0 Jul 20 13:17:04.092453 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.092383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"4ff26fdc17148000d01b14184bd9197c37d30c6860610abdbdb86bb9b1e929b5"} Jul 20 13:17:04.093810 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.093765 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3fe2c8d6a8514d87ee3bfa611989887" containerID="a6c1571c77d08de137cb83416181fff5fbd6021d9d4844f58fba3fd9de7c3ba1" exitCode=0 Jul 20 13:17:04.093876 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.093844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" event={"ID":"c3fe2c8d6a8514d87ee3bfa611989887","Type":"ContainerDied","Data":"a6c1571c77d08de137cb83416181fff5fbd6021d9d4844f58fba3fd9de7c3ba1"} Jul 20 13:17:04.097202 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.097182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"dc8f7d79096699083ee0119cc091e33c3550a5a142ee7ce8c1869d98a87e2fd2"} Jul 20 13:17:04.097275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.097239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"1d7c1fd1c4cdf34ddad8de6445e84369d229cbf8ed2eb029853e1a7c36aab5ff"} Jul 20 13:17:04.097275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.097271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"d12a3a319447a230ea88ffc77db6e4e4ac937e03775d7808b82756a6c847bdee"} Jul 20 13:17:04.097384 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.097305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"e12bc0a2f01bf19e8ae29d4886df30ad292b72636857364e42f34acaa02c0f51"} Jul 20 13:17:04.097384 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.097344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"166f6fc7cdffba015807cfaf0dd654e5cfa3a043759b322bba6221c9dfacadf6"} Jul 20 13:17:04.098519 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.098500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" event={"ID":"ea5539e4-8b05-40ee-b1c3-da73d8c3502f","Type":"ContainerStarted","Data":"564854b5b8d5ca07842986d27bdcfd051e8edbd099b82cfbff92dd963e2ffaf5"} Jul 20 13:17:04.099756 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.099736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drxgx" event={"ID":"f9cb3873-9970-4689-9e1f-423a90921969","Type":"ContainerStarted","Data":"0d59ebc6e5fd17c881f731face3c06432788f89fb563a9e97e5e79e40e6a03c7"} Jul 20 13:17:04.101095 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.101073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stgxm" event={"ID":"97144075-89d8-4105-a155-f224c2724c68","Type":"ContainerStarted","Data":"5ffd054198e70d6343681e59a726689658920e697190f26d80ee9ec5e4cac7f4"} Jul 20 13:17:04.108329 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.108283 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8dgkv" podStartSLOduration=3.437517028 podStartE2EDuration="23.108264748s" totalImagesPullingTime="19.67074772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:04.107952128 +0000 UTC m=+22.773891158" watchObservedRunningTime="2026-07-20 13:17:04.108264748 +0000 UTC m=+22.774203774" Jul 20 13:17:04.126822 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.125013 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-drxgx" podStartSLOduration=7.44216921 podStartE2EDuration="23.124965537s" totalImagesPullingTime="15.682796327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:04.12331281 +0000 UTC m=+22.789251841" watchObservedRunningTime="2026-07-20 13:17:04.124965537 +0000 UTC m=+22.790904561" Jul 20 13:17:04.157138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.157086 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-stgxm" podStartSLOduration=3.8804313 podStartE2EDuration="23.157070714s" totalImagesPullingTime="19.276639414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:04.140535638 +0000 UTC m=+22.806474693" watchObservedRunningTime="2026-07-20 13:17:04.157070714 +0000 UTC m=+22.823009744" Jul 20 13:17:04.876372 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:04.876314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:04.876731 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:04.876450 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:05.105339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.105301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tsbss" event={"ID":"8fc35a62-f9fe-46d9-8b66-3d5115c12503","Type":"ContainerStarted","Data":"57eb3b128cf1a5c6a32ca4f762156e74bc3afb669461959bc6c372d5253bcf44"} Jul 20 13:17:05.107179 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.107129 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 13:17:05.109145 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.108942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" event={"ID":"c3fe2c8d6a8514d87ee3bfa611989887","Type":"ContainerStarted","Data":"0597c957b17f4421dae836806be69c59ae0651b46bc5098d7b89a44ecfed01dc"} Jul 20 13:17:05.111682 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.111514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" event={"ID":"ea5539e4-8b05-40ee-b1c3-da73d8c3502f","Type":"ContainerStarted","Data":"83ebaeb1f15bbeff18e00fa021e681dadf8defb82a2665d84460997b8ca44ebc"} Jul 20 13:17:05.124489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.124449 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tsbss" podStartSLOduration=4.414263198 podStartE2EDuration="24.124432847s" totalImagesPullingTime="19.710169649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:05.124036256 +0000 UTC m=+23.789975289" watchObservedRunningTime="2026-07-20 13:17:05.124432847 +0000 UTC m=+23.790371878" Jul 20 13:17:05.141287 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.141244 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-97.ec2.internal" podStartSLOduration=23.141228251 podStartE2EDuration="23.141228251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:05.140842028 +0000 UTC m=+23.806781063" watchObservedRunningTime="2026-07-20 13:17:05.141228251 +0000 UTC m=+23.807167282" Jul 20 13:17:05.814015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.813908 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T13:17:05.107154183Z","UUID":"12cf00a8-c439-4e8b-8593-4f7f0afafc73","Handler":null,"Name":"","Endpoint":""} Jul 20 13:17:05.815498 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.815469 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 13:17:05.815631 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.815512 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 13:17:05.875506 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.875473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:05.875685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.875478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:05.875685 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:05.875628 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:05.876826 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:05.876382 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:05.985708 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.985316 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:17:05.986296 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:05.986273 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:17:06.119292 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:06.119231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"772b35a8e6a85cd4c04cfe6c8a2b11c318ce3b3c7cb5320e5707f3d9679974c7"} Jul 20 13:17:06.120008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:06.119967 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:17:06.120133 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:06.120042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8dgkv" Jul 20 13:17:06.876194 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:06.876124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:06.876327 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:06.876226 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:07.122959 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:07.122923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" event={"ID":"ea5539e4-8b05-40ee-b1c3-da73d8c3502f","Type":"ContainerStarted","Data":"4d985b4f109ec9e9333530c6cca521c374af1350e0ea1ca5ddfc73d445c8d8aa"} Jul 20 13:17:07.875795 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:07.875763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:07.875961 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:07.875897 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:07.875961 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:07.875945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:07.876105 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:07.876072 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:08.876448 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:08.876409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:08.876981 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:08.876535 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:09.131983 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.131708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" event={"ID":"80bdd741-0dab-4291-9470-e3a679e6b988","Type":"ContainerStarted","Data":"e71f3daf35c4009f7b4c5a18105961a5f64ba63a5b6c5b6e3c511c361fce6aa0"} Jul 20 13:17:09.132285 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.132005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:09.149760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.149222 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:09.163646 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.163380 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdqbl" podStartSLOduration=4.357558874 podStartE2EDuration="27.163341881s" totalImagesPullingTime="22.805783007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:07.158070695 +0000 UTC m=+25.824009727" watchObservedRunningTime="2026-07-20 13:17:09.163341881 +0000 UTC m=+27.829280915" Jul 20 13:17:09.196408 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.196201 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" podStartSLOduration=7.516202574 podStartE2EDuration="27.19618616s" totalImagesPullingTime="19.679983586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:09.164506152 +0000 UTC m=+27.830445183" watchObservedRunningTime="2026-07-20 13:17:09.19618616 +0000 UTC m=+27.862125191" Jul 20 13:17:09.875978 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.875907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:09.876161 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:09.876002 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:09.876161 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:09.876098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:09.876229 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:09.876184 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:10.135536 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.135500 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="18d4a62fed2943cf65be924fa03decdfce20db1149f8a5e1aa1744d5c3f4a6fa" exitCode=0 Jul 20 13:17:10.136116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.135550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"18d4a62fed2943cf65be924fa03decdfce20db1149f8a5e1aa1744d5c3f4a6fa"} Jul 20 13:17:10.136116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.135642 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 13:17:10.136116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.136073 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:10.156462 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.156433 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:10.875696 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:10.875661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:10.875883 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:10.875779 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:11.137986 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.137827 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 13:17:11.275306 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.275270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rvdwx"] Jul 20 13:17:11.275475 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.275449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:11.275616 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:11.275588 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:11.278267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.278206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kb7r"] Jul 20 13:17:11.278404 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.278318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:11.278471 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:11.278449 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:11.281222 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.281178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knxpg"] Jul 20 13:17:11.281318 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.281292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:11.281430 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:11.281403 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:11.417325 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:11.417261 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:12.142653 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:12.142626 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="f26e9c70b74697f7ed0819c276527429ee5ffc1e2c96784a1cb550e0b9fd2ec4" exitCode=0 Jul 20 13:17:12.143188 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:12.142697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"f26e9c70b74697f7ed0819c276527429ee5ffc1e2c96784a1cb550e0b9fd2ec4"} Jul 20 13:17:12.876541 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:12.876502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:12.876713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:12.876548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:12.876713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:12.876548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:12.876713 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:12.876650 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:12.876872 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:12.876723 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:12.876872 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:12.876821 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:13.147489 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:13.147461 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="11689048ce05420b9870e4993d13f889dbdec09b5f604acaf0997ca33f004241" exitCode=0 Jul 20 13:17:13.148082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:13.147555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"11689048ce05420b9870e4993d13f889dbdec09b5f604acaf0997ca33f004241"} Jul 20 13:17:14.604837 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.604783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:14.605227 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.604857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:14.605227 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.604938 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:14.605227 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.604953 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:14.605227 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.605003 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:46.604989888 +0000 UTC m=+65.270928898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:14.605227 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.605018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret podName:5e8945d1-09f8-44da-aeca-94ff385f58f8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:46.605012201 +0000 UTC m=+65.270951210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret") pod "global-pull-secret-syncer-rvdwx" (UID: "5e8945d1-09f8-44da-aeca-94ff385f58f8") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:14.705988 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.705954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:14.706286 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.706160 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:17:14.706286 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.706188 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:17:14.706286 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.706201 2575 projected.go:196] Error preparing data for projected volume kube-api-access-qw8dq for pod openshift-network-diagnostics/network-check-target-knxpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:14.706286 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.706267 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq podName:25b8106f-ae69-4949-aa29-357507d139ef nodeName:}" failed. No retries permitted until 2026-07-20 13:17:46.706247256 +0000 UTC m=+65.372186271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qw8dq" (UniqueName: "kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq") pod "network-check-target-knxpg" (UID: "25b8106f-ae69-4949-aa29-357507d139ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:14.876523 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.876443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:14.876683 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.876454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:14.876683 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.876574 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rvdwx" podUID="5e8945d1-09f8-44da-aeca-94ff385f58f8" Jul 20 13:17:14.876683 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:14.876462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:14.876853 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.876702 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:17:14.877498 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:14.876735 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knxpg" podUID="25b8106f-ae69-4949-aa29-357507d139ef" Jul 20 13:17:15.667879 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.667798 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-97.ec2.internal" event="NodeReady" Jul 20 13:17:15.668276 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.667931 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 13:17:15.711153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.711096 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:17:15.713530 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.713498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.717540 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.717513 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 13:17:15.717729 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.717705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58ttj\"" Jul 20 13:17:15.717829 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.717747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 13:17:15.718011 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.717996 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 13:17:15.723116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.723092 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 13:17:15.725749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.725723 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx"] Jul 20 13:17:15.728517 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.728493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.728644 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.728579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:17:15.731273 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.731252 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 20 13:17:15.731704 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.731684 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w5q64\"" Jul 20 13:17:15.731789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.731767 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 20 13:17:15.738938 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.738914 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8m4g"] Jul 20 13:17:15.741061 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.741041 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5hnnd"] Jul 20 13:17:15.741223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.741205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:15.743238 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.743216 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx"] Jul 20 13:17:15.743367 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.743311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:15.744753 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.744731 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 13:17:15.745061 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.745041 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:17:15.745231 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.745084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 13:17:15.746557 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.746537 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 13:17:15.746868 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.746843 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 13:17:15.746962 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.746850 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxzfq\"" Jul 20 13:17:15.746962 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.746880 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 13:17:15.755899 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.755875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hnnd"] Jul 20 13:17:15.756015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.755960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8m4g"] Jul 20 13:17:15.814024 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.813996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814024 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.814271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hr4\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814505 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.814505 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fb80410-b6cb-4568-aa83-bdea7c63f351-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.814505 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.814347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.914830 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.914830 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc55c466-02a5-40b0-bd25-230d7b714fcb-cert\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hr4\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-tmp-dir\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:15.914978 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.914979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915057 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:15.915045 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:16.415024615 +0000 UTC m=+35.080963629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:15.915099 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:15.915122 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:15.915190 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:16.415170913 +0000 UTC m=+35.081109944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmsl\" (UniqueName: \"kubernetes.io/projected/cc55c466-02a5-40b0-bd25-230d7b714fcb-kube-api-access-wzmsl\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fb80410-b6cb-4568-aa83-bdea7c63f351-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.915401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:15.915721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dv6f\" (UniqueName: \"kubernetes.io/projected/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-kube-api-access-6dv6f\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:15.915721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-config-volume\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:15.915874 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.915991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915962 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fb80410-b6cb-4568-aa83-bdea7c63f351-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:15.915991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.915989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.919464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.919278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.919464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.919298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.924419 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.924397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hr4\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:15.924419 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:15.924414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:16.016140 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc55c466-02a5-40b0-bd25-230d7b714fcb-cert\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:16.016339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-tmp-dir\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.016339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmsl\" (UniqueName: \"kubernetes.io/projected/cc55c466-02a5-40b0-bd25-230d7b714fcb-kube-api-access-wzmsl\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:16.016339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.016339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dv6f\" (UniqueName: \"kubernetes.io/projected/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-kube-api-access-6dv6f\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.016339 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-config-volume\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.016618 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.016596 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:16.016686 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.016673 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:16.516651494 +0000 UTC m=+35.182590517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:16.016817 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.016796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-tmp-dir\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.017031 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.017004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-config-volume\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.019293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.019269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc55c466-02a5-40b0-bd25-230d7b714fcb-cert\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:16.025930 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.025894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dv6f\" (UniqueName: \"kubernetes.io/projected/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-kube-api-access-6dv6f\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.026103 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.026082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmsl\" (UniqueName: \"kubernetes.io/projected/cc55c466-02a5-40b0-bd25-230d7b714fcb-kube-api-access-wzmsl\") pod \"ingress-canary-5hnnd\" (UID: \"cc55c466-02a5-40b0-bd25-230d7b714fcb\") " pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:16.063666 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.063637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hnnd" Jul 20 13:17:16.211005 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.210905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hnnd"] Jul 20 13:17:16.214966 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:17:16.214931 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc55c466_02a5_40b0_bd25_230d7b714fcb.slice/crio-64991f91835589dc6f296e26fc951d56e685f09b449243cc2a6d85fa19b64779 WatchSource:0}: Error finding container 64991f91835589dc6f296e26fc951d56e685f09b449243cc2a6d85fa19b64779: Status 404 returned error can't find the container with id 64991f91835589dc6f296e26fc951d56e685f09b449243cc2a6d85fa19b64779 Jul 20 13:17:16.418898 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.418848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:16.419057 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.419011 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:16.419117 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.419018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:16.419117 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.419087 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:16.419117 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.419101 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:16.419247 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.419093 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:17.419074797 +0000 UTC m=+36.085013826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:16.419247 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.419147 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:17.419136277 +0000 UTC m=+36.085075286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:16.520308 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.520222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:16.520483 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.520374 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:16.520483 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:16.520429 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:17.520415003 +0000 UTC m=+36.186354015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:16.876066 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.875985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:16.876066 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.876019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:16.876066 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.876024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:16.879153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879127 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:16.879270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879192 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:16.879270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:16.879270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879254 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:16.879455 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879192 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:17:16.879622 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:16.879606 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:17.159185 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:17.159151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hnnd" event={"ID":"cc55c466-02a5-40b0-bd25-230d7b714fcb","Type":"ContainerStarted","Data":"64991f91835589dc6f296e26fc951d56e685f09b449243cc2a6d85fa19b64779"} Jul 20 13:17:17.429996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:17.429922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:17.429996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:17.429968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:17.430230 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.430075 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:17.430230 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.430091 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:17.430230 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.430097 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:17.430230 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.430169 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:19.430150606 +0000 UTC m=+38.096089631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:17.430230 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.430186 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:19.430179549 +0000 UTC m=+38.096118558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:17.530978 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:17.530939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:17.531156 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.531139 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:17.531232 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:17.531212 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:19.531192042 +0000 UTC m=+38.197131056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:19.166727 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:19.166691 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="909040e8236e68f41bc266f7eb127ee94d3ec04a61497ea375c8197522a11db3" exitCode=0 Jul 20 13:17:19.167168 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:19.166774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"909040e8236e68f41bc266f7eb127ee94d3ec04a61497ea375c8197522a11db3"} Jul 20 13:17:19.449120 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:19.448878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:19.449120 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:19.449109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:19.449388 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.449040 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:19.449388 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.449225 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:23.449202508 +0000 UTC m=+42.115141519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:19.449388 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.449233 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:19.449388 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.449245 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:19.449388 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.449283 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:23.449271912 +0000 UTC m=+42.115210926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:19.549739 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:19.549704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:19.549888 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.549857 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:19.549927 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:19.549914 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:23.549898141 +0000 UTC m=+42.215837163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:20.172406 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:20.172369 2575 generic.go:356] "Generic (PLEG): container finished" podID="2f4c931e-a517-44a1-a18a-928eb98dcdf6" containerID="d05c4b976b24b951443e3d733a268789fbafafb1c1219396e27628f8f1b78330" exitCode=0 Jul 20 13:17:20.172822 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:20.172441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerDied","Data":"d05c4b976b24b951443e3d733a268789fbafafb1c1219396e27628f8f1b78330"} Jul 20 13:17:21.176716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:21.176681 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hnnd" event={"ID":"cc55c466-02a5-40b0-bd25-230d7b714fcb","Type":"ContainerStarted","Data":"92c96c606134983ef29b0a88ced148f11881e9531cf4956060890202316956eb"} Jul 20 13:17:21.180089 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:21.180058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" event={"ID":"2f4c931e-a517-44a1-a18a-928eb98dcdf6","Type":"ContainerStarted","Data":"6312ceed69e0f584315e8df0c6da88b44e507c1894816c81607db8c18c4bde2a"} Jul 20 13:17:21.201662 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:21.201608 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5hnnd" podStartSLOduration=1.510643483 podStartE2EDuration="6.201589516s" totalImagesPullingTime="4.690946033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:21.201584374 +0000 UTC m=+39.867523405" watchObservedRunningTime="2026-07-20 13:17:21.201589516 +0000 UTC m=+39.867528544" Jul 20 13:17:21.237504 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:21.237379 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ddzwq" podStartSLOduration=8.124316025 podStartE2EDuration="40.237343703s" totalImagesPullingTime="32.113027678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:21.236693217 +0000 UTC m=+39.902632248" watchObservedRunningTime="2026-07-20 13:17:21.237343703 +0000 UTC m=+39.903282735" Jul 20 13:17:23.483697 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:23.483657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:23.483715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.483813 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.483869 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.483883 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:31.48386632 +0000 UTC m=+50.149805329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.483886 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:23.484114 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.483927 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:31.483914322 +0000 UTC m=+50.149853331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:23.584298 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:23.584256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:23.584479 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.584384 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:23.584479 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:23.584435 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:31.584423013 +0000 UTC m=+50.250362023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:31.536173 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:31.536125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:31.536173 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:31.536174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:31.536697 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.536280 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:31.536697 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.536295 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:31.536697 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.536278 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:31.536697 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.536344 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:47.536330759 +0000 UTC m=+66.202269769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:31.536697 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.536372 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:47.536365687 +0000 UTC m=+66.202304696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:31.636793 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:31.636751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:31.636955 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.636902 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:31.636994 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:31.636966 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:17:47.636950637 +0000 UTC m=+66.302889645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:43.164561 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:43.164523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w2nmh" Jul 20 13:17:46.648652 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.648596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:46.649044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.648682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:17:46.651549 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.651510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:46.651677 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.651664 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:46.658949 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:46.658927 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:17:46.659038 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:46.659016 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:18:50.658993046 +0000 UTC m=+129.324932059 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : secret "metrics-daemon-secret" not found Jul 20 13:17:46.662635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.662616 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e8945d1-09f8-44da-aeca-94ff385f58f8-original-pull-secret\") pod \"global-pull-secret-syncer-rvdwx\" (UID: \"5e8945d1-09f8-44da-aeca-94ff385f58f8\") " pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:46.749854 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.749804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:46.753568 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.753544 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:46.763584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.763553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:46.773879 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.773846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8dq\" (UniqueName: \"kubernetes.io/projected/25b8106f-ae69-4949-aa29-357507d139ef-kube-api-access-qw8dq\") pod \"network-check-target-knxpg\" (UID: \"25b8106f-ae69-4949-aa29-357507d139ef\") " pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:46.889063 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.889020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rvdwx" Jul 20 13:17:46.899803 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.899743 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:46.907863 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:46.907841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:47.035708 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.035680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rvdwx"] Jul 20 13:17:47.038840 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:17:47.038816 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8945d1_09f8_44da_aeca_94ff385f58f8.slice/crio-d039d5eae42f5e179c39b78208faa24cb601c9971e3709fd6e462b2f71103722 WatchSource:0}: Error finding container d039d5eae42f5e179c39b78208faa24cb601c9971e3709fd6e462b2f71103722: Status 404 returned error can't find the container with id d039d5eae42f5e179c39b78208faa24cb601c9971e3709fd6e462b2f71103722 Jul 20 13:17:47.054081 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.054059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knxpg"] Jul 20 13:17:47.057023 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:17:47.057002 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b8106f_ae69_4949_aa29_357507d139ef.slice/crio-904f39a99ef0fd08c68ff5ac7e53cc1d077b35ee65b6bd69205bd070c7331630 WatchSource:0}: Error finding container 904f39a99ef0fd08c68ff5ac7e53cc1d077b35ee65b6bd69205bd070c7331630: Status 404 returned error can't find the container with id 904f39a99ef0fd08c68ff5ac7e53cc1d077b35ee65b6bd69205bd070c7331630 Jul 20 13:17:47.239118 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.239036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knxpg" event={"ID":"25b8106f-ae69-4949-aa29-357507d139ef","Type":"ContainerStarted","Data":"904f39a99ef0fd08c68ff5ac7e53cc1d077b35ee65b6bd69205bd070c7331630"} Jul 20 13:17:47.240035 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.240015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rvdwx" event={"ID":"5e8945d1-09f8-44da-aeca-94ff385f58f8","Type":"ContainerStarted","Data":"d039d5eae42f5e179c39b78208faa24cb601c9971e3709fd6e462b2f71103722"} Jul 20 13:17:47.556456 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.556345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:17:47.556456 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.556459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:17:47.556642 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.556504 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:17:47.556642 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.556523 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:17:47.556642 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.556547 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:17:47.556642 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.556578 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:18:19.556561694 +0000 UTC m=+98.222500702 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:17:47.556642 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.556591 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:18:19.556585301 +0000 UTC m=+98.222524309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:17:47.657017 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:47.656984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:17:47.657420 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.657097 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:47.657420 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:17:47.657147 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:18:19.657132878 +0000 UTC m=+98.323071887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:17:52.256288 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:52.255860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knxpg" event={"ID":"25b8106f-ae69-4949-aa29-357507d139ef","Type":"ContainerStarted","Data":"312fdf689e738a68454234d3d0ddeb493d65810a6e9a4306773488c497652bfd"} Jul 20 13:17:52.256288 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:52.256143 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:17:52.258553 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:52.258526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rvdwx" event={"ID":"5e8945d1-09f8-44da-aeca-94ff385f58f8","Type":"ContainerStarted","Data":"5a9a89fdd334331c26db21f37d5c450513b570e5e5bb23af2cb120433a4833d0"} Jul 20 13:17:52.283811 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:52.283753 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-knxpg" podStartSLOduration=65.212923938 podStartE2EDuration="1m10.283733541s" totalImagesPullingTime="5.070809603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:52.282944314 +0000 UTC m=+70.948883346" watchObservedRunningTime="2026-07-20 13:17:52.283733541 +0000 UTC m=+70.949672571" Jul 20 13:17:52.313517 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:17:52.313472 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rvdwx" podStartSLOduration=65.219512589 podStartE2EDuration="1m10.313452828s" totalImagesPullingTime="5.093940239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:52.313088467 +0000 UTC m=+70.979027498" watchObservedRunningTime="2026-07-20 13:17:52.313452828 +0000 UTC m=+70.979391856" Jul 20 13:18:10.197687 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.197573 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq"] Jul 20 13:18:10.200366 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.200340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.210030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.210005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 13:18:10.210153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.210033 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 20 13:18:10.210153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.210052 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 13:18:10.210153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.210054 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 13:18:10.210153 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.210008 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lfwn8\"" Jul 20 13:18:10.229641 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.229620 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq"] Jul 20 13:18:10.334752 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.334723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv6w\" (UniqueName: \"kubernetes.io/projected/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-kube-api-access-4zv6w\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.334992 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.334776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.435239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.435194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv6w\" (UniqueName: \"kubernetes.io/projected/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-kube-api-access-4zv6w\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.435461 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.435291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.437782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.437762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.450499 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.450438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv6w\" (UniqueName: \"kubernetes.io/projected/10e0d668-4403-4aaf-8d0d-61ddc42c06ca-kube-api-access-4zv6w\") pod \"managed-serviceaccount-addon-agent-d446777d4-zkqzq\" (UID: \"10e0d668-4403-4aaf-8d0d-61ddc42c06ca\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.519464 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.519429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" Jul 20 13:18:10.646254 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:10.646194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq"] Jul 20 13:18:10.650324 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:18:10.650301 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10e0d668_4403_4aaf_8d0d_61ddc42c06ca.slice/crio-0406687283aa3b1d048779d6fe9d7ca5399eee9623c84fe16b38cfdc9c809efd WatchSource:0}: Error finding container 0406687283aa3b1d048779d6fe9d7ca5399eee9623c84fe16b38cfdc9c809efd: Status 404 returned error can't find the container with id 0406687283aa3b1d048779d6fe9d7ca5399eee9623c84fe16b38cfdc9c809efd Jul 20 13:18:11.305976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:11.305937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" event={"ID":"10e0d668-4403-4aaf-8d0d-61ddc42c06ca","Type":"ContainerStarted","Data":"0406687283aa3b1d048779d6fe9d7ca5399eee9623c84fe16b38cfdc9c809efd"} Jul 20 13:18:14.314688 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:14.314651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" event={"ID":"10e0d668-4403-4aaf-8d0d-61ddc42c06ca","Type":"ContainerStarted","Data":"31b66aeceb96766fe66e56e39ec80e523dd5caba2045b35a3db235aa1c9bea75"} Jul 20 13:18:14.347760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:14.347710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" podStartSLOduration=1.554919518 podStartE2EDuration="4.347694052s" totalImagesPullingTime="2.792774534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:18:14.347169335 +0000 UTC m=+93.013108367" watchObservedRunningTime="2026-07-20 13:18:14.347694052 +0000 UTC m=+93.013633073" Jul 20 13:18:19.604274 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:19.604222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:19.604317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.604424 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.604440 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.604464 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.604479 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:19:23.604465949 +0000 UTC m=+162.270404958 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:18:19.604761 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.604540 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:19:23.604520126 +0000 UTC m=+162.270459139 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:18:19.705588 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:19.705554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:18:19.705755 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.705662 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:18:19.705755 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:19.705725 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:19:23.705711788 +0000 UTC m=+162.371650797 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:18:23.263877 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:23.263844 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-knxpg" Jul 20 13:18:50.737453 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:18:50.737395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:18:50.737954 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:50.737553 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:18:50.737954 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:18:50.737621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:20:52.737604336 +0000 UTC m=+251.403543345 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : secret "metrics-daemon-secret" not found Jul 20 13:19:18.725652 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:18.725599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-567c59b778-29msn" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" Jul 20 13:19:18.739797 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:18.739759 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" podUID="5fb80410-b6cb-4568-aa83-bdea7c63f351" Jul 20 13:19:18.754099 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:18.754066 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b8m4g" podUID="c3d5296d-3f5d-4f43-9633-a31bcf70b0ff" Jul 20 13:19:19.461709 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:19.461684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:19:19.461880 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:19.461684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:19:19.461880 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:19.461684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:19:19.902485 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:19.902429 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:19:23.681947 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:23.681886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:19:23.681947 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:23.681950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:19:23.682496 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.682057 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 13:19:23.682496 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.682065 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 13:19:23.682496 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.682159 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert podName:5fb80410-b6cb-4568-aa83-bdea7c63f351 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:25.682135077 +0000 UTC m=+284.348074104 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-j2tlx" (UID: "5fb80410-b6cb-4568-aa83-bdea7c63f351") : secret "networking-console-plugin-cert" not found Jul 20 13:19:23.682496 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.682165 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567c59b778-29msn: secret "image-registry-tls" not found Jul 20 13:19:23.682496 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.682208 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls podName:42b0d725-fe97-4137-b713-64ec782fa1a8 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:25.68219716 +0000 UTC m=+284.348136169 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls") pod "image-registry-567c59b778-29msn" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8") : secret "image-registry-tls" not found Jul 20 13:19:23.782527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:23.782487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:19:23.782695 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.782618 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:19:23.782695 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:19:23.782674 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls podName:c3d5296d-3f5d-4f43-9633-a31bcf70b0ff nodeName:}" failed. No retries permitted until 2026-07-20 13:21:25.782660714 +0000 UTC m=+284.448599722 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls") pod "dns-default-b8m4g" (UID: "c3d5296d-3f5d-4f43-9633-a31bcf70b0ff") : secret "dns-default-metrics-tls" not found Jul 20 13:19:30.876036 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:19:30.875999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:20:13.596058 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:13.596022 2575 generic.go:356] "Generic (PLEG): container finished" podID="10e0d668-4403-4aaf-8d0d-61ddc42c06ca" containerID="31b66aeceb96766fe66e56e39ec80e523dd5caba2045b35a3db235aa1c9bea75" exitCode=255 Jul 20 13:20:13.596480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:13.596072 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" event={"ID":"10e0d668-4403-4aaf-8d0d-61ddc42c06ca","Type":"ContainerDied","Data":"31b66aeceb96766fe66e56e39ec80e523dd5caba2045b35a3db235aa1c9bea75"} Jul 20 13:20:13.596480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:13.596367 2575 scope.go:117] "RemoveContainer" containerID="31b66aeceb96766fe66e56e39ec80e523dd5caba2045b35a3db235aa1c9bea75" Jul 20 13:20:14.599779 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:14.599747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d446777d4-zkqzq" event={"ID":"10e0d668-4403-4aaf-8d0d-61ddc42c06ca","Type":"ContainerStarted","Data":"96fc3074d284f7054992661eda28aab2992ed6f94b81706e95720748a075978c"} Jul 20 13:20:52.828548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:52.828509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:20:52.829025 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:52.828655 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:20:52.829025 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:52.828721 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs podName:dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2 nodeName:}" failed. No retries permitted until 2026-07-20 13:22:54.828705439 +0000 UTC m=+373.494644448 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs") pod "network-metrics-daemon-6kb7r" (UID: "dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2") : secret "metrics-daemon-secret" not found Jul 20 13:20:58.367211 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.367177 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-786b55b76d-4gts5"] Jul 20 13:20:58.369650 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.369632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.373830 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.373799 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 20 13:20:58.373830 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.373826 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.374052 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.374037 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rrmkx\"" Jul 20 13:20:58.374130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.374106 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 20 13:20:58.374245 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.374132 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 20 13:20:58.374245 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.374139 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.374480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.374467 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 20 13:20:58.395945 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.395915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-786b55b76d-4gts5"] Jul 20 13:20:58.459172 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.459142 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj"] Jul 20 13:20:58.461442 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.461427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:58.464501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.464474 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 20 13:20:58.464668 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.464517 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:20:58.464668 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.464533 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cbqtm\"" Jul 20 13:20:58.464851 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.464835 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 20 13:20:58.467589 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.467559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.467680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.467595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.467760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.467710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-default-certificate\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.467760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.467750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75zb\" (UniqueName: \"kubernetes.io/projected/6a56b5a7-6007-44c7-a47f-1bc6a516826d-kube-api-access-m75zb\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.467875 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.467800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-stats-auth\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.482956 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.482932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj"] Jul 20 13:20:58.568845 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.568805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-stats-auth\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.568845 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.568848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.568888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4sz\" (UniqueName: \"kubernetes.io/projected/6e93fdf1-2fbc-46d8-9487-f99d183821e0-kube-api-access-bt4sz\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.568909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.568930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:58.569030 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:58.569055 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:20:59.069039837 +0000 UTC m=+257.734978846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.569068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-default-certificate\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.569082 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:58.569079 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:20:59.069066369 +0000 UTC m=+257.735005378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:20:58.569349 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.569124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75zb\" (UniqueName: \"kubernetes.io/projected/6a56b5a7-6007-44c7-a47f-1bc6a516826d-kube-api-access-m75zb\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.571263 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.571244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-default-certificate\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.571337 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.571322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-stats-auth\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.578546 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.578527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75zb\" (UniqueName: \"kubernetes.io/projected/6a56b5a7-6007-44c7-a47f-1bc6a516826d-kube-api-access-m75zb\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:58.669669 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.669631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:58.669862 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.669682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4sz\" (UniqueName: \"kubernetes.io/projected/6e93fdf1-2fbc-46d8-9487-f99d183821e0-kube-api-access-bt4sz\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:58.669862 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:58.669794 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 13:20:58.669862 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:58.669860 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls podName:6e93fdf1-2fbc-46d8-9487-f99d183821e0 nodeName:}" failed. No retries permitted until 2026-07-20 13:20:59.169844789 +0000 UTC m=+257.835783797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ng6hj" (UID: "6e93fdf1-2fbc-46d8-9487-f99d183821e0") : secret "samples-operator-tls" not found Jul 20 13:20:58.682969 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:58.682941 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4sz\" (UniqueName: \"kubernetes.io/projected/6e93fdf1-2fbc-46d8-9487-f99d183821e0-kube-api-access-bt4sz\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:59.072954 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:59.072864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:59.072954 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:59.072904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:20:59.073145 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:59.073034 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:20:59.073145 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:59.073103 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:00.073084624 +0000 UTC m=+258.739023633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:20:59.073145 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:59.073121 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:00.073112797 +0000 UTC m=+258.739051820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:20:59.173708 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:20:59.173664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:20:59.173874 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:59.173813 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 13:20:59.173913 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:20:59.173878 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls podName:6e93fdf1-2fbc-46d8-9487-f99d183821e0 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:00.173863458 +0000 UTC m=+258.839802472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ng6hj" (UID: "6e93fdf1-2fbc-46d8-9487-f99d183821e0") : secret "samples-operator-tls" not found Jul 20 13:21:00.081673 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:00.081629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:00.081673 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:00.081671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:00.082085 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:00.081770 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:21:00.082085 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:00.081787 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:02.081773717 +0000 UTC m=+260.747712726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:21:00.082085 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:00.081821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:02.081807896 +0000 UTC m=+260.747746905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:21:00.182164 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:00.182118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:00.182326 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:00.182266 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 13:21:00.182326 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:00.182325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls podName:6e93fdf1-2fbc-46d8-9487-f99d183821e0 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:02.182310426 +0000 UTC m=+260.848249435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ng6hj" (UID: "6e93fdf1-2fbc-46d8-9487-f99d183821e0") : secret "samples-operator-tls" not found Jul 20 13:21:02.099108 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:02.099064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:02.099108 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:02.099104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:02.099545 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:02.099217 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:21:02.099545 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:02.099278 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.099263025 +0000 UTC m=+264.765202034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:21:02.099545 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:02.099296 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.099288987 +0000 UTC m=+264.765228000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:21:02.200285 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:02.200249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:02.200443 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:02.200420 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 13:21:02.200499 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:02.200490 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls podName:6e93fdf1-2fbc-46d8-9487-f99d183821e0 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.200476361 +0000 UTC m=+264.866415373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ng6hj" (UID: "6e93fdf1-2fbc-46d8-9487-f99d183821e0") : secret "samples-operator-tls" not found Jul 20 13:21:04.934670 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.934596 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6mvx7"] Jul 20 13:21:04.937617 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.937601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:04.945074 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.945052 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 13:21:04.945367 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.945335 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 13:21:04.945433 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.945418 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 13:21:04.945614 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.945601 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 13:21:04.948473 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.948460 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fctt2\"" Jul 20 13:21:04.961119 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:04.961098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mvx7"] Jul 20 13:21:05.022333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.022295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.022527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.022377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.022527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.022478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpq5p\" (UniqueName: \"kubernetes.io/projected/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-api-access-rpq5p\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.022527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.022518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-crio-socket\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.022621 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.022609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-data-volume\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123341 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-data-volume\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123341 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpq5p\" (UniqueName: \"kubernetes.io/projected/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-api-access-rpq5p\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-crio-socket\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.123529 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-crio-socket\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.123573 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.123527 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.123765 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.123591 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.623575613 +0000 UTC m=+264.289514621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.123765 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.123645 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.623628981 +0000 UTC m=+264.289567995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : configmap "kube-rbac-proxy" not found Jul 20 13:21:05.123765 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.123690 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-data-volume\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.137414 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.137382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpq5p\" (UniqueName: \"kubernetes.io/projected/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-api-access-rpq5p\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.627222 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.627166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.627222 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:05.627235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:05.627480 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.627322 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.627480 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.627342 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.627480 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.627393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.627379151 +0000 UTC m=+265.293318160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : configmap "kube-rbac-proxy" not found Jul 20 13:21:05.627480 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:05.627424 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.627409335 +0000 UTC m=+265.293348345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.132086 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.132049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:06.132515 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.132136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:06.132515 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.132209 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:14.13219209 +0000 UTC m=+272.798131098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:21:06.132515 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.132244 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:21:06.132515 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.132284 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:14.132272948 +0000 UTC m=+272.798211960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:21:06.232763 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.232727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:06.232911 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.232870 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 13:21:06.232953 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.232928 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls podName:6e93fdf1-2fbc-46d8-9487-f99d183821e0 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:14.232910582 +0000 UTC m=+272.898849596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ng6hj" (UID: "6e93fdf1-2fbc-46d8-9487-f99d183821e0") : secret "samples-operator-tls" not found Jul 20 13:21:06.595917 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.595823 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4kbqx"] Jul 20 13:21:06.598817 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.598799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.602170 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.602143 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 20 13:21:06.603544 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.603520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 20 13:21:06.603633 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.603583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mvqjf\"" Jul 20 13:21:06.603694 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.603650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 20 13:21:06.603743 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.603702 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 20 13:21:06.608633 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.608610 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4kbqx"] Jul 20 13:21:06.636170 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.636131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:06.636347 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.636243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:06.636347 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.636263 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:06.636347 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.636328 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.636309416 +0000 UTC m=+267.302248424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : configmap "kube-rbac-proxy" not found Jul 20 13:21:06.636492 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.636348 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.636492 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:06.636425 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.636411611 +0000 UTC m=+267.302350620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.737401 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.737343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vdw\" (UniqueName: \"kubernetes.io/projected/3c4613b9-ddfd-46ab-a680-b52c3057dbba-kube-api-access-72vdw\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.737575 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.737427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-cabundle\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.737575 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.737471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-key\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.838400 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.838348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vdw\" (UniqueName: \"kubernetes.io/projected/3c4613b9-ddfd-46ab-a680-b52c3057dbba-kube-api-access-72vdw\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.838525 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.838424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-cabundle\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.838525 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.838459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-key\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.839038 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.839012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-cabundle\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.840778 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.840757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c4613b9-ddfd-46ab-a680-b52c3057dbba-signing-key\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.847707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.847646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vdw\" (UniqueName: \"kubernetes.io/projected/3c4613b9-ddfd-46ab-a680-b52c3057dbba-kube-api-access-72vdw\") pod \"service-ca-58598695f7-4kbqx\" (UID: \"3c4613b9-ddfd-46ab-a680-b52c3057dbba\") " pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:06.908444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:06.908402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-4kbqx" Jul 20 13:21:07.038029 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:07.037996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4kbqx"] Jul 20 13:21:07.041120 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:07.041097 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4613b9_ddfd_46ab_a680_b52c3057dbba.slice/crio-87c591a7fe4d3952e8b95e45abd0761db613155147897656922862b820cb24f0 WatchSource:0}: Error finding container 87c591a7fe4d3952e8b95e45abd0761db613155147897656922862b820cb24f0: Status 404 returned error can't find the container with id 87c591a7fe4d3952e8b95e45abd0761db613155147897656922862b820cb24f0 Jul 20 13:21:07.494718 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:07.494692 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drxgx_f9cb3873-9970-4689-9e1f-423a90921969/dns-node-resolver/0.log" Jul 20 13:21:07.723842 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:07.723808 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-4kbqx" event={"ID":"3c4613b9-ddfd-46ab-a680-b52c3057dbba","Type":"ContainerStarted","Data":"87c591a7fe4d3952e8b95e45abd0761db613155147897656922862b820cb24f0"} Jul 20 13:21:08.656982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:08.656882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:08.656982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:08.656951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:08.657458 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:08.657056 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:08.657458 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:08.657137 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.657120048 +0000 UTC m=+271.323059063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:08.657458 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:08.657068 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:08.657458 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:08.657209 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.657191216 +0000 UTC m=+271.323130244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : configmap "kube-rbac-proxy" not found Jul 20 13:21:08.894282 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:08.894245 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stgxm_97144075-89d8-4105-a155-f224c2724c68/node-ca/0.log" Jul 20 13:21:09.494564 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:09.494538 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hnnd_cc55c466-02a5-40b0-bd25-230d7b714fcb/serve-healthcheck-canary/0.log" Jul 20 13:21:09.731077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:09.731039 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-4kbqx" event={"ID":"3c4613b9-ddfd-46ab-a680-b52c3057dbba","Type":"ContainerStarted","Data":"5050c7e0040b99eec304dd4bfbf8f43b32cddd4c1379638681f4aa8bd94faa0c"} Jul 20 13:21:09.748058 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:09.747949 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-4kbqx" podStartSLOduration=1.7804999910000001 podStartE2EDuration="3.747931531s" totalImagesPullingTime="1.96743154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:09.746865259 +0000 UTC m=+268.412804289" watchObservedRunningTime="2026-07-20 13:21:09.747931531 +0000 UTC m=+268.413870592" Jul 20 13:21:12.690135 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:12.690096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:12.690602 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:12.690150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:12.690602 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:12.690220 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:12.690602 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:12.690268 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:12.690602 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:12.690282 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.690269663 +0000 UTC m=+279.356208675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : configmap "kube-rbac-proxy" not found Jul 20 13:21:12.690602 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:12.690346 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls podName:a986f2ae-eac9-4fe7-8f37-c18b44ed11f1 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.690329288 +0000 UTC m=+279.356268299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mvx7" (UID: "a986f2ae-eac9-4fe7-8f37-c18b44ed11f1") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:14.203615 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.203571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:14.204111 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.203658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:14.204111 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:14.203750 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 20 13:21:14.204111 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:14.203761 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:30.203742599 +0000 UTC m=+288.869681609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : configmap references non-existent config key: service-ca.crt Jul 20 13:21:14.204111 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:14.203790 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs podName:6a56b5a7-6007-44c7-a47f-1bc6a516826d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:30.203779276 +0000 UTC m=+288.869718285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs") pod "router-default-786b55b76d-4gts5" (UID: "6a56b5a7-6007-44c7-a47f-1bc6a516826d") : secret "router-metrics-certs-default" not found Jul 20 13:21:14.304375 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.304305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:14.306751 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.306723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e93fdf1-2fbc-46d8-9487-f99d183821e0-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ng6hj\" (UID: \"6e93fdf1-2fbc-46d8-9487-f99d183821e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:14.370579 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.370534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" Jul 20 13:21:14.512342 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.512247 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj"] Jul 20 13:21:14.751409 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:14.751372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" event={"ID":"6e93fdf1-2fbc-46d8-9487-f99d183821e0","Type":"ContainerStarted","Data":"975844e3d02742eff814ee24d17e58d464bfd682281470b71d6a9b62f24c06b0"} Jul 20 13:21:16.759958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:16.759833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" event={"ID":"6e93fdf1-2fbc-46d8-9487-f99d183821e0","Type":"ContainerStarted","Data":"683549d69cbd212da93cd2dedba392e5d4e61ff828da66073c50bea0275c00ce"} Jul 20 13:21:16.759958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:16.759899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" event={"ID":"6e93fdf1-2fbc-46d8-9487-f99d183821e0","Type":"ContainerStarted","Data":"c1689dbd0b55c8071f4790f4fead2a3a9f6136fd37e099648f9a402245ff1233"} Jul 20 13:21:16.779262 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:16.779204 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ng6hj" podStartSLOduration=16.735634794 podStartE2EDuration="18.779181169s" totalImagesPullingTime="2.043546375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:16.778412263 +0000 UTC m=+275.444351294" watchObservedRunningTime="2026-07-20 13:21:16.779181169 +0000 UTC m=+275.445120201" Jul 20 13:21:20.758071 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.758017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:20.758492 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.758086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:20.758635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.758615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:20.760363 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.760333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a986f2ae-eac9-4fe7-8f37-c18b44ed11f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mvx7\" (UID: \"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1\") " pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:20.846969 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.846937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mvx7" Jul 20 13:21:20.985347 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:20.985314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mvx7"] Jul 20 13:21:20.986040 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:20.986007 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda986f2ae_eac9_4fe7_8f37_c18b44ed11f1.slice/crio-f7437a311b6df000220089fb7d7533a46cf337108b327580a515a4ebfd8203be WatchSource:0}: Error finding container f7437a311b6df000220089fb7d7533a46cf337108b327580a515a4ebfd8203be: Status 404 returned error can't find the container with id f7437a311b6df000220089fb7d7533a46cf337108b327580a515a4ebfd8203be Jul 20 13:21:21.785773 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:21.784456 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mvx7" event={"ID":"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1","Type":"ContainerStarted","Data":"f7437a311b6df000220089fb7d7533a46cf337108b327580a515a4ebfd8203be"} Jul 20 13:21:22.462498 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:22.462436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" podUID="5fb80410-b6cb-4568-aa83-bdea7c63f351" Jul 20 13:21:22.462498 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:22.462453 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b8m4g" podUID="c3d5296d-3f5d-4f43-9633-a31bcf70b0ff" Jul 20 13:21:22.462498 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:22.462485 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-567c59b778-29msn" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" Jul 20 13:21:22.788112 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:22.788033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:21:22.788709 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:22.788033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:22.788709 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:22.788033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:25.703980 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.703939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:21:25.704519 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.704007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:25.706885 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.706854 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"image-registry-567c59b778-29msn\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:25.707008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.706919 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fb80410-b6cb-4568-aa83-bdea7c63f351-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-j2tlx\" (UID: \"5fb80410-b6cb-4568-aa83-bdea7c63f351\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:21:25.793561 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.793522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58ttj\"" Jul 20 13:21:25.793765 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.793522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w5q64\"" Jul 20 13:21:25.800016 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.799986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" Jul 20 13:21:25.800165 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.800018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:25.804699 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.804673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:25.807551 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:25.807518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3d5296d-3f5d-4f43-9633-a31bcf70b0ff-metrics-tls\") pod \"dns-default-b8m4g\" (UID: \"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff\") " pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:26.013166 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.013127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx"] Jul 20 13:21:26.016246 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:26.016218 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb80410_b6cb_4568_aa83_bdea7c63f351.slice/crio-a6cd4ae351c743304815ef24949ebc0ba11ec9d69e048642e376a06608a6661b WatchSource:0}: Error finding container a6cd4ae351c743304815ef24949ebc0ba11ec9d69e048642e376a06608a6661b: Status 404 returned error can't find the container with id a6cd4ae351c743304815ef24949ebc0ba11ec9d69e048642e376a06608a6661b Jul 20 13:21:26.029615 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.029581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:21:26.032898 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:26.032871 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b0d725_fe97_4137_b713_64ec782fa1a8.slice/crio-a8c636cbc66fd4a5a466cf4bfdd762dd19a90af25e03b03f451a8f3630dabb7f WatchSource:0}: Error finding container a8c636cbc66fd4a5a466cf4bfdd762dd19a90af25e03b03f451a8f3630dabb7f: Status 404 returned error can't find the container with id a8c636cbc66fd4a5a466cf4bfdd762dd19a90af25e03b03f451a8f3630dabb7f Jul 20 13:21:26.093509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.093488 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:21:26.099885 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.099855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:26.230404 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.230308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8m4g"] Jul 20 13:21:26.233701 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:26.233670 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d5296d_3f5d_4f43_9633_a31bcf70b0ff.slice/crio-207784b299697025d15f38b3e2a23a8e3f80fc22334c39379ebcb3dee96e3c7a WatchSource:0}: Error finding container 207784b299697025d15f38b3e2a23a8e3f80fc22334c39379ebcb3dee96e3c7a: Status 404 returned error can't find the container with id 207784b299697025d15f38b3e2a23a8e3f80fc22334c39379ebcb3dee96e3c7a Jul 20 13:21:26.805291 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.805251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mvx7" event={"ID":"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1","Type":"ContainerStarted","Data":"59c0d59ec95664b3e4ad713fbcab3e7548dfeaf678c096529b0ec3905f93b6c3"} Jul 20 13:21:26.806884 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.806850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" event={"ID":"5fb80410-b6cb-4568-aa83-bdea7c63f351","Type":"ContainerStarted","Data":"a6cd4ae351c743304815ef24949ebc0ba11ec9d69e048642e376a06608a6661b"} Jul 20 13:21:26.808385 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.808306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8m4g" event={"ID":"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff","Type":"ContainerStarted","Data":"207784b299697025d15f38b3e2a23a8e3f80fc22334c39379ebcb3dee96e3c7a"} Jul 20 13:21:26.810914 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.810873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567c59b778-29msn" event={"ID":"42b0d725-fe97-4137-b713-64ec782fa1a8","Type":"ContainerStarted","Data":"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a"} Jul 20 13:21:26.811053 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.810925 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567c59b778-29msn" event={"ID":"42b0d725-fe97-4137-b713-64ec782fa1a8","Type":"ContainerStarted","Data":"a8c636cbc66fd4a5a466cf4bfdd762dd19a90af25e03b03f451a8f3630dabb7f"} Jul 20 13:21:26.811170 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.811117 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:26.837927 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:26.837520 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-567c59b778-29msn" podStartSLOduration=284.837496535 podStartE2EDuration="4m44.837496535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:26.835734914 +0000 UTC m=+285.501673945" watchObservedRunningTime="2026-07-20 13:21:26.837496535 +0000 UTC m=+285.503435568" Jul 20 13:21:27.815704 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:27.815615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" event={"ID":"5fb80410-b6cb-4568-aa83-bdea7c63f351","Type":"ContainerStarted","Data":"80da7274a7bfe0b5a7ee374d21643701653e00250beb5addd2fbd012d0d3ef3b"} Jul 20 13:21:27.835997 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:27.835933 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-j2tlx" podStartSLOduration=269.381875989 podStartE2EDuration="4m30.835914186s" totalImagesPullingTime="1.454038197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:27.834078606 +0000 UTC m=+286.500017637" watchObservedRunningTime="2026-07-20 13:21:27.835914186 +0000 UTC m=+286.501853194" Jul 20 13:21:30.246331 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.246298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:30.246726 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.246385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:30.247031 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.247009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a56b5a7-6007-44c7-a47f-1bc6a516826d-service-ca-bundle\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:30.248616 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.248595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a56b5a7-6007-44c7-a47f-1bc6a516826d-metrics-certs\") pod \"router-default-786b55b76d-4gts5\" (UID: \"6a56b5a7-6007-44c7-a47f-1bc6a516826d\") " pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:30.478742 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.478700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:30.615447 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.615415 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-786b55b76d-4gts5"] Jul 20 13:21:30.618649 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:30.618620 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a56b5a7_6007_44c7_a47f_1bc6a516826d.slice/crio-75b9071d243caca50314c2e2b79ddc51d02ee856d8ebe415c3e39fa160a75474 WatchSource:0}: Error finding container 75b9071d243caca50314c2e2b79ddc51d02ee856d8ebe415c3e39fa160a75474: Status 404 returned error can't find the container with id 75b9071d243caca50314c2e2b79ddc51d02ee856d8ebe415c3e39fa160a75474 Jul 20 13:21:30.829223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.828950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8m4g" event={"ID":"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff","Type":"ContainerStarted","Data":"bf67cc35c3cc4025ad1a7bffeda4e3cac16ca915bab56e9148af77c130f73cb6"} Jul 20 13:21:30.829223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.829026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8m4g" event={"ID":"c3d5296d-3f5d-4f43-9633-a31bcf70b0ff","Type":"ContainerStarted","Data":"e944873d662c672feb9c90a0e863c34147d1d9f06715c84f41a3c51975a9f109"} Jul 20 13:21:30.829223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.829060 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:30.830741 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.830710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-786b55b76d-4gts5" event={"ID":"6a56b5a7-6007-44c7-a47f-1bc6a516826d","Type":"ContainerStarted","Data":"abd49253af953852426d2836b0cb7161f919e347878f52d46b9ece7858565896"} Jul 20 13:21:30.830885 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.830763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-786b55b76d-4gts5" event={"ID":"6a56b5a7-6007-44c7-a47f-1bc6a516826d","Type":"ContainerStarted","Data":"75b9071d243caca50314c2e2b79ddc51d02ee856d8ebe415c3e39fa160a75474"} Jul 20 13:21:30.833082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.832909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mvx7" event={"ID":"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1","Type":"ContainerStarted","Data":"51f5d0e7017887b5e86185c7e23c977e312bf9c447a1c62d25692bf1312f6b73"} Jul 20 13:21:30.876661 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.876610 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-786b55b76d-4gts5" podStartSLOduration=32.876591967 podStartE2EDuration="32.876591967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:30.875465243 +0000 UTC m=+289.541404286" watchObservedRunningTime="2026-07-20 13:21:30.876591967 +0000 UTC m=+289.542530998" Jul 20 13:21:30.876831 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:30.876812 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b8m4g" podStartSLOduration=252.049272646 podStartE2EDuration="4m15.8768035s" totalImagesPullingTime="3.827530854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:30.850278252 +0000 UTC m=+289.516217283" watchObservedRunningTime="2026-07-20 13:21:30.8768035 +0000 UTC m=+289.542742531" Jul 20 13:21:31.479422 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.479324 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:31.482526 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.482493 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:31.709809 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.709756 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:21:31.716345 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.716308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.719750 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.719710 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 13:21:31.720010 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.719994 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 13:21:31.720308 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 13:21:31.720424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720329 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 13:21:31.720424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720288 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 13:21:31.720549 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720298 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6jkhh\"" Jul 20 13:21:31.720706 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 13:21:31.720791 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.720730 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 13:21:31.729916 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.729854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:21:31.836710 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.836681 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:31.838142 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.838120 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-786b55b76d-4gts5" Jul 20 13:21:31.861704 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.861899 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.861899 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.861899 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.861899 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.862123 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.861955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962551 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962854 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.962854 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.962823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.963280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.963255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.963912 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.963889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.964274 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.964220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.966247 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.966210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.967081 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.967047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:31.973217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:31.973193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56\") pod \"console-6967644dd4-84scx\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:32.028216 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:32.028140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:33.842319 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:33.842293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:21:33.844755 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:33.844721 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bb457d_c2df_4114_b5bb_a5f6aba941b6.slice/crio-a10512a245557cc031f057b6e3a845d7ec5ea7c12b1a48a15cd60f2afaaa7ce7 WatchSource:0}: Error finding container a10512a245557cc031f057b6e3a845d7ec5ea7c12b1a48a15cd60f2afaaa7ce7: Status 404 returned error can't find the container with id a10512a245557cc031f057b6e3a845d7ec5ea7c12b1a48a15cd60f2afaaa7ce7 Jul 20 13:21:33.846707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:33.846661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mvx7" event={"ID":"a986f2ae-eac9-4fe7-8f37-c18b44ed11f1","Type":"ContainerStarted","Data":"2804768c65d5782810a652244cf2fa8a1446df38e36fae8c38d7739705431b69"} Jul 20 13:21:33.867239 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:33.867175 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6mvx7" podStartSLOduration=17.283591338 podStartE2EDuration="29.867151946s" totalImagesPullingTime="12.583560608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:33.866099778 +0000 UTC m=+292.532038819" watchObservedRunningTime="2026-07-20 13:21:33.867151946 +0000 UTC m=+292.533090978" Jul 20 13:21:33.876930 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:21:33.876889 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6kb7r" podUID="dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2" Jul 20 13:21:34.851438 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:34.851366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967644dd4-84scx" event={"ID":"80bb457d-c2df-4114-b5bb-a5f6aba941b6","Type":"ContainerStarted","Data":"a10512a245557cc031f057b6e3a845d7ec5ea7c12b1a48a15cd60f2afaaa7ce7"} Jul 20 13:21:37.867614 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:37.867572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967644dd4-84scx" event={"ID":"80bb457d-c2df-4114-b5bb-a5f6aba941b6","Type":"ContainerStarted","Data":"c918ad2cde2b36e8a9845e5e858e09734eefb8eea5e540bf1cad15b498118e35"} Jul 20 13:21:37.888187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:37.888135 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6967644dd4-84scx" podStartSLOduration=3.410645185 podStartE2EDuration="6.88811965s" totalImagesPullingTime="3.477474465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:37.887205951 +0000 UTC m=+296.553144982" watchObservedRunningTime="2026-07-20 13:21:37.88811965 +0000 UTC m=+296.554058681" Jul 20 13:21:40.838376 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:40.838325 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8m4g" Jul 20 13:21:41.081864 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.081823 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:21:41.284581 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.284541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:21:41.284729 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.284620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.292506 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.292481 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 13:21:41.442658 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz4n\" (UniqueName: \"kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.442967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.442815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543736 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz4n\" (UniqueName: \"kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543736 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543736 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543736 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.543917 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.543750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.544298 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.544271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.544426 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.544342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.544539 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.544516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.544638 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.544621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.545986 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.545965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.546137 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.546117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.553683 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.553652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz4n\" (UniqueName: \"kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n\") pod \"console-66bbc9946b-5sbvp\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.593738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.593697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:41.761870 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.761840 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 13:21:41.787880 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.787851 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:21:41.790936 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:41.790908 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd206b5_12e8_45ed_a349_16881332165b.slice/crio-d008f6f8994083ffd1420383dda09a5882bc0777a20bf5ed8ce523e53940b8e6 WatchSource:0}: Error finding container d008f6f8994083ffd1420383dda09a5882bc0777a20bf5ed8ce523e53940b8e6: Status 404 returned error can't find the container with id d008f6f8994083ffd1420383dda09a5882bc0777a20bf5ed8ce523e53940b8e6 Jul 20 13:21:41.883253 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:41.883219 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bbc9946b-5sbvp" event={"ID":"1cd206b5-12e8-45ed-a349-16881332165b","Type":"ContainerStarted","Data":"d008f6f8994083ffd1420383dda09a5882bc0777a20bf5ed8ce523e53940b8e6"} Jul 20 13:21:42.115647 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:42.028259 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:42.115647 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:42.028350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:21:42.115647 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:42.029658 2575 patch_prober.go:28] interesting pod/console-6967644dd4-84scx container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Jul 20 13:21:42.115647 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:42.029710 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6967644dd4-84scx" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Jul 20 13:21:42.888645 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:42.888601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bbc9946b-5sbvp" event={"ID":"1cd206b5-12e8-45ed-a349-16881332165b","Type":"ContainerStarted","Data":"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d"} Jul 20 13:21:43.729809 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.729744 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66bbc9946b-5sbvp" podStartSLOduration=2.729724577 podStartE2EDuration="2.729724577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:42.911017186 +0000 UTC m=+301.576956216" watchObservedRunningTime="2026-07-20 13:21:43.729724577 +0000 UTC m=+302.395663608" Jul 20 13:21:43.730072 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.730058 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x9b9x"] Jul 20 13:21:43.814716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.814677 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj"] Jul 20 13:21:43.814907 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.814817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.818501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.818478 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 13:21:43.818684 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.818587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 13:21:43.818800 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.818782 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 13:21:43.818861 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.818839 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 13:21:43.819284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.819260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzg4g\"" Jul 20 13:21:43.819518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.819502 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 13:21:43.819963 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.819935 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 13:21:43.832609 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.832582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj"] Jul 20 13:21:43.832749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.832627 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:43.835568 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.835548 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9nd2h\"" Jul 20 13:21:43.835719 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.835631 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 20 13:21:43.835824 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.835807 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 20 13:21:43.965784 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.965754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-root\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.965784 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.965789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-wtmp\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.965815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbdm\" (UniqueName: \"kubernetes.io/projected/3416af86-934d-430e-88e4-fb8bc068f9f4-kube-api-access-tmbdm\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.965848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.965874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-metrics-client-ca\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-tls\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-sys\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kl5b\" (UniqueName: \"kubernetes.io/projected/14397bd9-1bec-4120-af3f-40aaed40255b-kube-api-access-5kl5b\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-textfile\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3416af86-934d-430e-88e4-fb8bc068f9f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:43.966333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:43.966240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.066755 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.066652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.066755 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.066699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-metrics-client-ca\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-tls\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067221 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-sys\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067221 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-sys\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067221 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kl5b\" (UniqueName: \"kubernetes.io/projected/14397bd9-1bec-4120-af3f-40aaed40255b-kube-api-access-5kl5b\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-textfile\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3416af86-934d-430e-88e4-fb8bc068f9f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-metrics-client-ca\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-root\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-wtmp\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbdm\" (UniqueName: \"kubernetes.io/projected/3416af86-934d-430e-88e4-fb8bc068f9f4-kube-api-access-tmbdm\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.067849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067650 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-root\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-textfile\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.067849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.067802 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-wtmp\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.068565 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.068539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3416af86-934d-430e-88e4-fb8bc068f9f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.069612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.069584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.069721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.069662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14397bd9-1bec-4120-af3f-40aaed40255b-node-exporter-tls\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.069826 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.069804 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.070034 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.070013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3416af86-934d-430e-88e4-fb8bc068f9f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.080693 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.080664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kl5b\" (UniqueName: \"kubernetes.io/projected/14397bd9-1bec-4120-af3f-40aaed40255b-kube-api-access-5kl5b\") pod \"node-exporter-x9b9x\" (UID: \"14397bd9-1bec-4120-af3f-40aaed40255b\") " pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.080939 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.080922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbdm\" (UniqueName: \"kubernetes.io/projected/3416af86-934d-430e-88e4-fb8bc068f9f4-kube-api-access-tmbdm\") pod \"openshift-state-metrics-79f6958cdf-hshbj\" (UID: \"3416af86-934d-430e-88e4-fb8bc068f9f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.126497 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.126466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x9b9x" Jul 20 13:21:44.142184 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.142154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" Jul 20 13:21:44.168810 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:44.168786 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14397bd9_1bec_4120_af3f_40aaed40255b.slice/crio-aab3f4993e0442081642dbdb7acc23ac6bc3c185ec7f882e3a0d8566407a1ee1 WatchSource:0}: Error finding container aab3f4993e0442081642dbdb7acc23ac6bc3c185ec7f882e3a0d8566407a1ee1: Status 404 returned error can't find the container with id aab3f4993e0442081642dbdb7acc23ac6bc3c185ec7f882e3a0d8566407a1ee1 Jul 20 13:21:44.170909 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.170885 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:21:44.319537 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.319464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj"] Jul 20 13:21:44.322166 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:44.322140 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3416af86_934d_430e_88e4_fb8bc068f9f4.slice/crio-810c9db4663d0626cddd63974b48069eecde6fc740b6b654217cebfd381a569c WatchSource:0}: Error finding container 810c9db4663d0626cddd63974b48069eecde6fc740b6b654217cebfd381a569c: Status 404 returned error can't find the container with id 810c9db4663d0626cddd63974b48069eecde6fc740b6b654217cebfd381a569c Jul 20 13:21:44.875933 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.875844 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:21:44.888184 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.888149 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 13:21:44.909168 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.909127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9b9x" event={"ID":"14397bd9-1bec-4120-af3f-40aaed40255b","Type":"ContainerStarted","Data":"aab3f4993e0442081642dbdb7acc23ac6bc3c185ec7f882e3a0d8566407a1ee1"} Jul 20 13:21:44.909168 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.909167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" event={"ID":"3416af86-934d-430e-88e4-fb8bc068f9f4","Type":"ContainerStarted","Data":"36a649e4d278e556ff8fb7398b4fe7ac3f2aa760f231846ee638e668639ab0d6"} Jul 20 13:21:44.909454 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.909190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" event={"ID":"3416af86-934d-430e-88e4-fb8bc068f9f4","Type":"ContainerStarted","Data":"fbe8676180efe08188e22bab89ea7ddca995a54b8a3c511956f6b9a40af3e1bc"} Jul 20 13:21:44.909454 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.909206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" event={"ID":"3416af86-934d-430e-88e4-fb8bc068f9f4","Type":"ContainerStarted","Data":"810c9db4663d0626cddd63974b48069eecde6fc740b6b654217cebfd381a569c"} Jul 20 13:21:44.909454 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.909281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:44.918314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.917761 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 20 13:21:44.918314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.917789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 20 13:21:44.918314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.917914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 20 13:21:44.919509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.919200 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 13:21:44.920217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920022 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 20 13:21:44.920217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 20 13:21:44.920217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920171 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 20 13:21:44.920460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2lp4r\"" Jul 20 13:21:44.920460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920396 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 20 13:21:44.920769 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.920748 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 20 13:21:44.926757 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:44.926729 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 20 13:21:45.076668 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-out\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.076998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzk4\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-kube-api-access-bnzk4\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.077036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.077063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-web-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077634 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.077116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.077634 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.077154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178437 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-out\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzk4\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-kube-api-access-bnzk4\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.178838 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.179171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.178859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-web-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.179171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.179128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.179564 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.179442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.180237 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.180179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4042774-51a5-4116-9ed0-c9bb3bf2f710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.182138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.182109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-web-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.182493 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.182434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-out\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.182493 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.182456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.182666 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.182596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.183014 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.182966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.183555 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.183513 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.184006 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.183984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.186424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.184272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.186424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.184751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4042774-51a5-4116-9ed0-c9bb3bf2f710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.193296 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.193269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzk4\" (UniqueName: \"kubernetes.io/projected/c4042774-51a5-4116-9ed0-c9bb3bf2f710-kube-api-access-bnzk4\") pod \"alertmanager-main-0\" (UID: \"c4042774-51a5-4116-9ed0-c9bb3bf2f710\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.221454 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.221372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 20 13:21:45.503863 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.503820 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 13:21:45.507452 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:45.507421 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4042774_51a5_4116_9ed0_c9bb3bf2f710.slice/crio-03afc88a80eb34aa3e5c104b2913baacf44b2b97efe819d33d8f177c89e2d3e7 WatchSource:0}: Error finding container 03afc88a80eb34aa3e5c104b2913baacf44b2b97efe819d33d8f177c89e2d3e7: Status 404 returned error can't find the container with id 03afc88a80eb34aa3e5c104b2913baacf44b2b97efe819d33d8f177c89e2d3e7 Jul 20 13:21:45.806719 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.806634 2575 patch_prober.go:28] interesting pod/image-registry-567c59b778-29msn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 13:21:45.806719 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.806691 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-567c59b778-29msn" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 13:21:45.905327 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:45.905292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"03afc88a80eb34aa3e5c104b2913baacf44b2b97efe819d33d8f177c89e2d3e7"} Jul 20 13:21:46.841610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.841577 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp"] Jul 20 13:21:46.844520 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.844489 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.847703 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.847681 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 20 13:21:46.848388 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 20 13:21:46.848680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848565 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eu9l8hn1bdodm\"" Jul 20 13:21:46.848680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848621 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 20 13:21:46.848680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 20 13:21:46.848680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848691 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rbsr2\"" Jul 20 13:21:46.848993 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.848712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 20 13:21:46.862747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.862718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp"] Jul 20 13:21:46.911713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.911657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" event={"ID":"3416af86-934d-430e-88e4-fb8bc068f9f4","Type":"ContainerStarted","Data":"7f47dfb21e50e426c10a1191994f764fc8759e7bfd6c094e923ea7dbc2913f02"} Jul 20 13:21:46.913508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.913478 2575 generic.go:356] "Generic (PLEG): container finished" podID="14397bd9-1bec-4120-af3f-40aaed40255b" containerID="bdbe5def4ef1507eceb2445d54373cac8a5177f70fc84e9c8005e7105e44bb52" exitCode=0 Jul 20 13:21:46.913658 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.913522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9b9x" event={"ID":"14397bd9-1bec-4120-af3f-40aaed40255b","Type":"ContainerDied","Data":"bdbe5def4ef1507eceb2445d54373cac8a5177f70fc84e9c8005e7105e44bb52"} Jul 20 13:21:46.935273 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.935173 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hshbj" podStartSLOduration=2.302647626 podStartE2EDuration="3.935149595s" totalImagesPullingTime="1.632501969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:46.934379614 +0000 UTC m=+305.600318649" watchObservedRunningTime="2026-07-20 13:21:46.935149595 +0000 UTC m=+305.601088627" Jul 20 13:21:46.996486 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996689 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-grpc-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996689 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlb49\" (UniqueName: \"kubernetes.io/projected/61359bfc-4ca0-4ffb-ae57-72b567c4c100-kube-api-access-hlb49\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996812 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61359bfc-4ca0-4ffb-ae57-72b567c4c100-metrics-client-ca\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996940 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:46.996940 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:46.996889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097413 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61359bfc-4ca0-4ffb-ae57-72b567c4c100-metrics-client-ca\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097413 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097556 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097556 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097556 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-grpc-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097718 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097718 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlb49\" (UniqueName: \"kubernetes.io/projected/61359bfc-4ca0-4ffb-ae57-72b567c4c100-kube-api-access-hlb49\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.097718 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.097693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.098322 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.098159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61359bfc-4ca0-4ffb-ae57-72b567c4c100-metrics-client-ca\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.100910 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.100730 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.100910 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.100851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.101116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.100979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.101116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.101015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-grpc-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.101116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.101090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.101260 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.101121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/61359bfc-4ca0-4ffb-ae57-72b567c4c100-secret-thanos-querier-tls\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.107555 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.107532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlb49\" (UniqueName: \"kubernetes.io/projected/61359bfc-4ca0-4ffb-ae57-72b567c4c100-kube-api-access-hlb49\") pod \"thanos-querier-7bb4f6cc59-tqgsp\" (UID: \"61359bfc-4ca0-4ffb-ae57-72b567c4c100\") " pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.155420 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.155384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:47.322974 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.322941 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp"] Jul 20 13:21:47.327059 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:47.327032 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61359bfc_4ca0_4ffb_ae57_72b567c4c100.slice/crio-6d0e3be5bfc6ad4cac28e55599f918d0e5c8ae60ac1cf1ddea1f1768a97b03ad WatchSource:0}: Error finding container 6d0e3be5bfc6ad4cac28e55599f918d0e5c8ae60ac1cf1ddea1f1768a97b03ad: Status 404 returned error can't find the container with id 6d0e3be5bfc6ad4cac28e55599f918d0e5c8ae60ac1cf1ddea1f1768a97b03ad Jul 20 13:21:47.820324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.820292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:21:47.920825 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.920767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9b9x" event={"ID":"14397bd9-1bec-4120-af3f-40aaed40255b","Type":"ContainerStarted","Data":"1ff74ad97b73fac14ed734e014708362a81a15dab1ff1aacceafabfa79e97d16"} Jul 20 13:21:47.920825 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.920833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x9b9x" event={"ID":"14397bd9-1bec-4120-af3f-40aaed40255b","Type":"ContainerStarted","Data":"490656578b2d969650dd1de2020701531861b0e6f3ed5d167716e5f2e826b568"} Jul 20 13:21:47.922584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.922556 2575 generic.go:356] "Generic (PLEG): container finished" podID="c4042774-51a5-4116-9ed0-c9bb3bf2f710" containerID="c1eccdd7e705c4e92b3f7d7c6822eaee5ced72a7a07257330d08c0ffc037a249" exitCode=0 Jul 20 13:21:47.922726 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.922642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerDied","Data":"c1eccdd7e705c4e92b3f7d7c6822eaee5ced72a7a07257330d08c0ffc037a249"} Jul 20 13:21:47.924208 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.924183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"6d0e3be5bfc6ad4cac28e55599f918d0e5c8ae60ac1cf1ddea1f1768a97b03ad"} Jul 20 13:21:47.946422 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:47.946334 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x9b9x" podStartSLOduration=3.053873739 podStartE2EDuration="4.946311946s" totalImagesPullingTime="1.797398854s" totalInitContainerRuntime="95.039353ms" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:47.946181915 +0000 UTC m=+306.612120947" watchObservedRunningTime="2026-07-20 13:21:47.946311946 +0000 UTC m=+306.612250978" Jul 20 13:21:48.831521 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:48.831481 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:21:48.869856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:48.869823 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:21:48.872258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:48.872237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:48.887564 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:48.887536 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:21:49.018983 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.018948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.018983 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.018983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8mn\" (UniqueName: \"kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.019508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.019103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.019508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.019154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.019508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.019209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.019508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.019232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.019508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.019368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8mn\" (UniqueName: \"kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120841 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120841 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120841 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.120841 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.120807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.121555 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.121504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.121676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.121570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.121848 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.121819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.122074 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.122033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.123802 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.123778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.124179 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.124082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.130214 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.130178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8mn\" (UniqueName: \"kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn\") pod \"console-866b57b788-bzpv8\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.182164 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.182130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:49.927307 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.927266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:21:49.930080 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:21:49.930019 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6ebb0b_dca9_453c_85a2_9ed5bf163895.slice/crio-9c4c240ae609c60cd08e6a3d3b9232abdd0524a1a6fe7f22a931b95053ff45b0 WatchSource:0}: Error finding container 9c4c240ae609c60cd08e6a3d3b9232abdd0524a1a6fe7f22a931b95053ff45b0: Status 404 returned error can't find the container with id 9c4c240ae609c60cd08e6a3d3b9232abdd0524a1a6fe7f22a931b95053ff45b0 Jul 20 13:21:49.941499 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.941385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"653581be29bf1a0fffc4a3d1fe7b8d7b5ab9b656d2014b573a01621abc3cb28b"} Jul 20 13:21:49.941499 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.941466 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"818caf0dc367c3ff492287fcae3b2d097945dd6df70a5caaef97965a023edab2"} Jul 20 13:21:49.943722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.943679 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"2347e3b55925718767f8ac86a8e53a0807d0fbc8d4ac06db52929f3f462e3dd0"} Jul 20 13:21:49.943976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:49.943757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"a695429189ce92b12f50f6bc34c26942f8921dbfaa82b111d5b405a012358294"} Jul 20 13:21:50.951094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.951052 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"35f057499aa7ee332319def6254f0259f9bb56908d9f1cf988e8125963f76128"} Jul 20 13:21:50.951569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.951166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"ab80cf823327b16d9994c67dd8412e13b090da558f448d32b9e21146edb9bf90"} Jul 20 13:21:50.951569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.951210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"3d090aac647d54b1cab1467fd75e51a54e05acc1ea55f4bb356aaae849e15a7c"} Jul 20 13:21:50.953232 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.953206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"a2c7e4db2b61d49b98aa5ffdbf85ddd3a4eaf8e741730fef87be68f4c6c7f9b3"} Jul 20 13:21:50.954650 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.954624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b57b788-bzpv8" event={"ID":"cc6ebb0b-dca9-453c-85a2-9ed5bf163895","Type":"ContainerStarted","Data":"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911"} Jul 20 13:21:50.954765 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.954670 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b57b788-bzpv8" event={"ID":"cc6ebb0b-dca9-453c-85a2-9ed5bf163895","Type":"ContainerStarted","Data":"9c4c240ae609c60cd08e6a3d3b9232abdd0524a1a6fe7f22a931b95053ff45b0"} Jul 20 13:21:50.979810 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:50.979764 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866b57b788-bzpv8" podStartSLOduration=2.979748236 podStartE2EDuration="2.979748236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:50.97751157 +0000 UTC m=+309.643450602" watchObservedRunningTime="2026-07-20 13:21:50.979748236 +0000 UTC m=+309.645687267" Jul 20 13:21:51.331371 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.331318 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:21:51.593983 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.593895 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:51.593983 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.593951 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:51.598803 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.598779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:51.961896 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.961860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4042774-51a5-4116-9ed0-c9bb3bf2f710","Type":"ContainerStarted","Data":"fe3ae4009f6ca655396f02d68930a5074716c76189fabaf12e3017e4f7a17015"} Jul 20 13:21:51.964698 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.964670 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"73319da3b448a91b37a5f650116a6a99d68c6d3a211b257568d01afc42aca35e"} Jul 20 13:21:51.964835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.964726 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"ac7c9069c3f72f46b876e9a3c3b5ac9b39be508c90709c3ef9aa3450c52df428"} Jul 20 13:21:51.964835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.964752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" event={"ID":"61359bfc-4ca0-4ffb-ae57-72b567c4c100","Type":"ContainerStarted","Data":"82d385aced2b4ba5ab4b57bf24c6cadb0681842a93fabc36e48af440c6681b80"} Jul 20 13:21:51.968543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.968520 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:21:51.993836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:51.993786 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.533005249 podStartE2EDuration="7.993769264s" totalImagesPullingTime="4.438628183s" totalInitContainerRuntime="22.135832ms" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:51.992339773 +0000 UTC m=+310.658278804" watchObservedRunningTime="2026-07-20 13:21:51.993769264 +0000 UTC m=+310.659708294" Jul 20 13:21:52.049003 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:52.048955 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" podStartSLOduration=2.641867295 podStartE2EDuration="6.048938436s" totalImagesPullingTime="3.407071141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:52.026549236 +0000 UTC m=+310.692488272" watchObservedRunningTime="2026-07-20 13:21:52.048938436 +0000 UTC m=+310.714877503" Jul 20 13:21:52.968590 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:52.968559 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:58.979171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:58.979142 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bb4f6cc59-tqgsp" Jul 20 13:21:59.183025 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:59.182990 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:59.183208 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:59.183048 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:59.187849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:59.187822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:21:59.999669 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:21:59.999643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:22:00.059270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:00.059233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:22:13.857035 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:13.856967 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6967644dd4-84scx" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerName="console" containerID="cri-o://c918ad2cde2b36e8a9845e5e858e09734eefb8eea5e540bf1cad15b498118e35" gracePeriod=15 Jul 20 13:22:14.049823 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.049790 2575 generic.go:356] "Generic (PLEG): container finished" podID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerID="c918ad2cde2b36e8a9845e5e858e09734eefb8eea5e540bf1cad15b498118e35" exitCode=0 Jul 20 13:22:14.049975 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.049845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967644dd4-84scx" event={"ID":"80bb457d-c2df-4114-b5bb-a5f6aba941b6","Type":"ContainerDied","Data":"c918ad2cde2b36e8a9845e5e858e09734eefb8eea5e540bf1cad15b498118e35"} Jul 20 13:22:14.098069 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.098038 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:22:14.147591 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.147748 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147642 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.147842 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.147893 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147883 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.148013 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147951 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.148013 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.147988 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config\") pod \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\" (UID: \"80bb457d-c2df-4114-b5bb-a5f6aba941b6\") " Jul 20 13:22:14.148122 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.148012 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:14.148222 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.148202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca" (OuterVolumeSpecName: "service-ca") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:14.148460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.148430 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-oauth-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:14.148460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.148454 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-service-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:14.148649 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.148495 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config" (OuterVolumeSpecName: "console-config") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:14.149951 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.149926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:14.150445 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.150424 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:14.150517 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.150485 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56" (OuterVolumeSpecName: "kube-api-access-zqm56") pod "80bb457d-c2df-4114-b5bb-a5f6aba941b6" (UID: "80bb457d-c2df-4114-b5bb-a5f6aba941b6"). InnerVolumeSpecName "kube-api-access-zqm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:22:14.249410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.249335 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm56\" (UniqueName: \"kubernetes.io/projected/80bb457d-c2df-4114-b5bb-a5f6aba941b6-kube-api-access-zqm56\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:14.249410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.249402 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:14.249410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.249413 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-oauth-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:14.249410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:14.249422 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80bb457d-c2df-4114-b5bb-a5f6aba941b6-console-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:15.055407 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.055376 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967644dd4-84scx" Jul 20 13:22:15.055953 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.055410 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967644dd4-84scx" event={"ID":"80bb457d-c2df-4114-b5bb-a5f6aba941b6","Type":"ContainerDied","Data":"a10512a245557cc031f057b6e3a845d7ec5ea7c12b1a48a15cd60f2afaaa7ce7"} Jul 20 13:22:15.055953 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.055465 2575 scope.go:117] "RemoveContainer" containerID="c918ad2cde2b36e8a9845e5e858e09734eefb8eea5e540bf1cad15b498118e35" Jul 20 13:22:15.084020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.083991 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:22:15.089780 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.089753 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6967644dd4-84scx"] Jul 20 13:22:15.881715 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:15.881680 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" path="/var/lib/kubelet/pods/80bb457d-c2df-4114-b5bb-a5f6aba941b6/volumes" Jul 20 13:22:16.351565 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.351477 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-567c59b778-29msn" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerName="registry" containerID="cri-o://806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a" gracePeriod=30 Jul 20 13:22:16.600181 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.600156 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:22:16.671408 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671366 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671423 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671499 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99hr4\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671525 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671561 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671589 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671858 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671630 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.671858 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671667 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted\") pod \"42b0d725-fe97-4137-b713-64ec782fa1a8\" (UID: \"42b0d725-fe97-4137-b713-64ec782fa1a8\") " Jul 20 13:22:16.672012 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.671975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:16.672494 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.672456 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:16.674002 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.673960 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:16.674119 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.674066 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:16.674119 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.674108 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:22:16.674237 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.674126 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:22:16.674297 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.674283 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4" (OuterVolumeSpecName: "kube-api-access-99hr4") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "kube-api-access-99hr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:22:16.681502 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.681475 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "42b0d725-fe97-4137-b713-64ec782fa1a8" (UID: "42b0d725-fe97-4137-b713-64ec782fa1a8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:22:16.772872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772833 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hr4\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-kube-api-access-99hr4\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.772872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772861 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-trusted-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.772872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772871 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-certificates\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.772872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772885 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-image-registry-private-configuration\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.773138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772895 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b0d725-fe97-4137-b713-64ec782fa1a8-installation-pull-secrets\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.773138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772903 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b0d725-fe97-4137-b713-64ec782fa1a8-ca-trust-extracted\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.773138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772912 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-registry-tls\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:16.773138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:16.772920 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b0d725-fe97-4137-b713-64ec782fa1a8-bound-sa-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:17.069240 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.069161 2575 generic.go:356] "Generic (PLEG): container finished" podID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerID="806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a" exitCode=0 Jul 20 13:22:17.069240 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.069212 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567c59b778-29msn" Jul 20 13:22:17.069444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.069253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567c59b778-29msn" event={"ID":"42b0d725-fe97-4137-b713-64ec782fa1a8","Type":"ContainerDied","Data":"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a"} Jul 20 13:22:17.069444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.069332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567c59b778-29msn" event={"ID":"42b0d725-fe97-4137-b713-64ec782fa1a8","Type":"ContainerDied","Data":"a8c636cbc66fd4a5a466cf4bfdd762dd19a90af25e03b03f451a8f3630dabb7f"} Jul 20 13:22:17.069444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.069375 2575 scope.go:117] "RemoveContainer" containerID="806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a" Jul 20 13:22:17.077910 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.077890 2575 scope.go:117] "RemoveContainer" containerID="806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a" Jul 20 13:22:17.078159 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:22:17.078135 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a\": container with ID starting with 806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a not found: ID does not exist" containerID="806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a" Jul 20 13:22:17.078234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.078164 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a"} err="failed to get container status \"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a\": rpc error: code = NotFound desc = could not find container \"806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a\": container with ID starting with 806e2b3a40cf9390f29fec09a92271bc8a78c62b1c06001f3caa0b52db3bde8a not found: ID does not exist" Jul 20 13:22:17.096162 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.096138 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:22:17.099716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.099695 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-567c59b778-29msn"] Jul 20 13:22:17.880666 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:17.880632 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" path="/var/lib/kubelet/pods/42b0d725-fe97-4137-b713-64ec782fa1a8/volumes" Jul 20 13:22:25.079713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.079671 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66bbc9946b-5sbvp" podUID="1cd206b5-12e8-45ed-a349-16881332165b" containerName="console" containerID="cri-o://6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d" gracePeriod=15 Jul 20 13:22:25.327349 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.327323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:22:25.446982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.446948 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhz4n\" (UniqueName: \"kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447155 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447155 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447050 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447155 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447071 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447334 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447318 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert\") pod \"1cd206b5-12e8-45ed-a349-16881332165b\" (UID: \"1cd206b5-12e8-45ed-a349-16881332165b\") " Jul 20 13:22:25.447569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447526 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config" (OuterVolumeSpecName: "console-config") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:25.447569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447535 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca" (OuterVolumeSpecName: "service-ca") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:25.447569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:25.447931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447798 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-console-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.447931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447807 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:22:25.447931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447818 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-trusted-ca-bundle\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.447931 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.447834 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-service-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.449218 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.449194 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n" (OuterVolumeSpecName: "kube-api-access-qhz4n") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "kube-api-access-qhz4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:22:25.449349 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.449326 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:25.449546 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.449390 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1cd206b5-12e8-45ed-a349-16881332165b" (UID: "1cd206b5-12e8-45ed-a349-16881332165b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:22:25.548244 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.548197 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.548244 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.548237 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cd206b5-12e8-45ed-a349-16881332165b-console-oauth-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.548244 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.548247 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cd206b5-12e8-45ed-a349-16881332165b-oauth-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:25.548244 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:25.548256 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhz4n\" (UniqueName: \"kubernetes.io/projected/1cd206b5-12e8-45ed-a349-16881332165b-kube-api-access-qhz4n\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:22:26.105079 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.105042 2575 generic.go:356] "Generic (PLEG): container finished" podID="1cd206b5-12e8-45ed-a349-16881332165b" containerID="6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d" exitCode=0 Jul 20 13:22:26.105548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.105090 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bbc9946b-5sbvp" Jul 20 13:22:26.105548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.105125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bbc9946b-5sbvp" event={"ID":"1cd206b5-12e8-45ed-a349-16881332165b","Type":"ContainerDied","Data":"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d"} Jul 20 13:22:26.105548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.105173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bbc9946b-5sbvp" event={"ID":"1cd206b5-12e8-45ed-a349-16881332165b","Type":"ContainerDied","Data":"d008f6f8994083ffd1420383dda09a5882bc0777a20bf5ed8ce523e53940b8e6"} Jul 20 13:22:26.105548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.105193 2575 scope.go:117] "RemoveContainer" containerID="6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d" Jul 20 13:22:26.113480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.113463 2575 scope.go:117] "RemoveContainer" containerID="6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d" Jul 20 13:22:26.113766 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:22:26.113749 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d\": container with ID starting with 6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d not found: ID does not exist" containerID="6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d" Jul 20 13:22:26.113815 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.113776 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d"} err="failed to get container status \"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d\": rpc error: code = NotFound desc = could not find container \"6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d\": container with ID starting with 6e990eaaa2a88b60ef89a4b935c8c7171d82c1b04aec4a5b22885e36ee496a3d not found: ID does not exist" Jul 20 13:22:26.126621 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.122991 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:22:26.129323 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:26.129296 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66bbc9946b-5sbvp"] Jul 20 13:22:27.880028 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:27.879996 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd206b5-12e8-45ed-a349-16881332165b" path="/var/lib/kubelet/pods/1cd206b5-12e8-45ed-a349-16881332165b/volumes" Jul 20 13:22:54.892098 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:54.892059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:22:54.894408 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:54.894382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2-metrics-certs\") pod \"network-metrics-daemon-6kb7r\" (UID: \"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2\") " pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:22:55.079772 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:55.079735 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:22:55.087271 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:55.087245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6kb7r" Jul 20 13:22:55.222462 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:55.220861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6kb7r"] Jul 20 13:22:55.225647 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:22:55.225619 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaa569d_e15d_44f2_b1cd_13ddbbcfdff2.slice/crio-f73c76e1dac5c7a2e17b6374bad98b3373d9cfb878ec1c3b9b5523e68393ae3d WatchSource:0}: Error finding container f73c76e1dac5c7a2e17b6374bad98b3373d9cfb878ec1c3b9b5523e68393ae3d: Status 404 returned error can't find the container with id f73c76e1dac5c7a2e17b6374bad98b3373d9cfb878ec1c3b9b5523e68393ae3d Jul 20 13:22:56.229005 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:56.228966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kb7r" event={"ID":"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2","Type":"ContainerStarted","Data":"f73c76e1dac5c7a2e17b6374bad98b3373d9cfb878ec1c3b9b5523e68393ae3d"} Jul 20 13:22:57.234011 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:57.233975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kb7r" event={"ID":"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2","Type":"ContainerStarted","Data":"b7d80670b720ca6ac99db255e9b39f5ae4baf2653c06c2a03b145daec1041241"} Jul 20 13:22:57.234430 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:57.234025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6kb7r" event={"ID":"dbaa569d-e15d-44f2-b1cd-13ddbbcfdff2","Type":"ContainerStarted","Data":"613d0b8f5acc84f871443c426441dba9bcf39ba75055a73d1ebe5bb16ae2e812"} Jul 20 13:22:57.254303 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:22:57.254228 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6kb7r" podStartSLOduration=373.92944155 podStartE2EDuration="6m15.25421286s" totalImagesPullingTime="1.32477131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:22:57.251869546 +0000 UTC m=+375.917808575" watchObservedRunningTime="2026-07-20 13:22:57.25421286 +0000 UTC m=+375.920151889" Jul 20 13:23:08.289008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.288975 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57ff4874b9-ghvd7"] Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289258 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerName="registry" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289268 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerName="registry" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289281 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd206b5-12e8-45ed-a349-16881332165b" containerName="console" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289286 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd206b5-12e8-45ed-a349-16881332165b" containerName="console" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289295 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerName="console" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289303 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerName="console" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289379 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b0d725-fe97-4137-b713-64ec782fa1a8" containerName="registry" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289389 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="80bb457d-c2df-4114-b5bb-a5f6aba941b6" containerName="console" Jul 20 13:23:08.289518 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.289394 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd206b5-12e8-45ed-a349-16881332165b" containerName="console" Jul 20 13:23:08.292844 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.292826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.296181 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.296157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jdzzk\"" Jul 20 13:23:08.296286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.296207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 20 13:23:08.297442 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.297427 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 20 13:23:08.297515 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.297468 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 20 13:23:08.298684 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.298656 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 20 13:23:08.300103 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.300079 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 20 13:23:08.306789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.306773 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 20 13:23:08.312952 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.312932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57ff4874b9-ghvd7"] Jul 20 13:23:08.410737 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410847 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410847 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2snx\" (UniqueName: \"kubernetes.io/projected/67494da6-17a8-4f54-bc5d-d081f86d1407-kube-api-access-p2snx\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410913 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-serving-certs-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410913 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-federate-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410980 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410980 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.410980 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.410977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-metrics-client-ca\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511395 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2snx\" (UniqueName: \"kubernetes.io/projected/67494da6-17a8-4f54-bc5d-d081f86d1407-kube-api-access-p2snx\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511608 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-serving-certs-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511608 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-federate-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511608 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511608 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511608 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-metrics-client-ca\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511857 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.511857 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.511680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.512333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.512256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-serving-certs-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.512471 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.512380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-metrics-client-ca\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.512561 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.512538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.514410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.514378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.514636 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.514612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-telemeter-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.514696 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.514620 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-federate-client-tls\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.514696 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.514623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/67494da6-17a8-4f54-bc5d-d081f86d1407-secret-telemeter-client\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.522576 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.522558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2snx\" (UniqueName: \"kubernetes.io/projected/67494da6-17a8-4f54-bc5d-d081f86d1407-kube-api-access-p2snx\") pod \"telemeter-client-57ff4874b9-ghvd7\" (UID: \"67494da6-17a8-4f54-bc5d-d081f86d1407\") " pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.602672 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.602566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" Jul 20 13:23:08.741886 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:08.741847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57ff4874b9-ghvd7"] Jul 20 13:23:08.746298 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:23:08.746251 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67494da6_17a8_4f54_bc5d_d081f86d1407.slice/crio-937213f301757925fab62d6e7718626acce0739ef2cfd97924bec1d0d974e531 WatchSource:0}: Error finding container 937213f301757925fab62d6e7718626acce0739ef2cfd97924bec1d0d974e531: Status 404 returned error can't find the container with id 937213f301757925fab62d6e7718626acce0739ef2cfd97924bec1d0d974e531 Jul 20 13:23:09.279822 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:09.279788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" event={"ID":"67494da6-17a8-4f54-bc5d-d081f86d1407","Type":"ContainerStarted","Data":"937213f301757925fab62d6e7718626acce0739ef2cfd97924bec1d0d974e531"} Jul 20 13:23:11.290332 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:11.290208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" event={"ID":"67494da6-17a8-4f54-bc5d-d081f86d1407","Type":"ContainerStarted","Data":"e57b458aea39710e28924dadaf889069b8dfc5bdb66a26ed47dd16965f1e14fc"} Jul 20 13:23:11.290332 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:11.290309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" event={"ID":"67494da6-17a8-4f54-bc5d-d081f86d1407","Type":"ContainerStarted","Data":"053465b24fe8d88fd2fd776893aa191343fc2575fddaa15c74441e2cb3f66b32"} Jul 20 13:23:11.290868 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:11.290340 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" event={"ID":"67494da6-17a8-4f54-bc5d-d081f86d1407","Type":"ContainerStarted","Data":"41c546b73129a4c7106be0fac6ee1315e3b58077276b201f2a8489703b368ea3"} Jul 20 13:23:11.328380 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:11.328306 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57ff4874b9-ghvd7" podStartSLOduration=1.057634294 podStartE2EDuration="3.328290873s" totalImagesPullingTime="2.270656579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:23:11.326988589 +0000 UTC m=+389.992927621" watchObservedRunningTime="2026-07-20 13:23:11.328290873 +0000 UTC m=+389.994229902" Jul 20 13:23:12.058654 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.058617 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:12.061686 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.061661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.086096 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.086066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:12.144060 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144060 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrsk\" (UniqueName: \"kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.144286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.144262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245519 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245717 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245717 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245717 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245717 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrsk\" (UniqueName: \"kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.245906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.245792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.246389 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.246340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.246514 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.246406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.246514 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.246499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.246588 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.246507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.248673 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.248649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.248765 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.248744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.254860 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.254836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrsk\" (UniqueName: \"kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk\") pod \"console-7784476d9f-dzgw9\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.371125 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.371039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:12.505797 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:12.505767 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:12.508046 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:23:12.508012 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45b02f9_fb4c_4d6e_a9e0_86e933c6f81b.slice/crio-96c79af39c709439d5fcc1ef5caabc546ea7ee9f91993356239334b15e4fabcc WatchSource:0}: Error finding container 96c79af39c709439d5fcc1ef5caabc546ea7ee9f91993356239334b15e4fabcc: Status 404 returned error can't find the container with id 96c79af39c709439d5fcc1ef5caabc546ea7ee9f91993356239334b15e4fabcc Jul 20 13:23:13.303955 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:13.303914 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784476d9f-dzgw9" event={"ID":"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b","Type":"ContainerStarted","Data":"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0"} Jul 20 13:23:13.304136 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:13.303975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784476d9f-dzgw9" event={"ID":"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b","Type":"ContainerStarted","Data":"96c79af39c709439d5fcc1ef5caabc546ea7ee9f91993356239334b15e4fabcc"} Jul 20 13:23:13.326509 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:13.326461 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7784476d9f-dzgw9" podStartSLOduration=1.326443737 podStartE2EDuration="1.326443737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:23:13.32470534 +0000 UTC m=+391.990644372" watchObservedRunningTime="2026-07-20 13:23:13.326443737 +0000 UTC m=+391.992382767" Jul 20 13:23:19.566545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.566509 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:19.594103 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.594069 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:23:19.597798 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.597773 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.614127 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.614100 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:23:19.708015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.707967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.708246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.708166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7s92\" (UniqueName: \"kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809069 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809069 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s92\" (UniqueName: \"kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.809555 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.809378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.810038 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.810009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.810225 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.810011 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.810313 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.810013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.810313 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.810023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.811863 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.811842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.811918 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.811876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.817956 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.817893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s92\" (UniqueName: \"kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92\") pod \"console-95c875b-ctvnl\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:19.907968 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:19.907933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:20.048127 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:20.048091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:23:20.052005 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:23:20.051972 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c826a0_10f0_4a6a_97af_44f84390b431.slice/crio-ff6b18cb143b46a8ff1dc72d31277dae3671d2e6dc05006e3b097540bfdd99ab WatchSource:0}: Error finding container ff6b18cb143b46a8ff1dc72d31277dae3671d2e6dc05006e3b097540bfdd99ab: Status 404 returned error can't find the container with id ff6b18cb143b46a8ff1dc72d31277dae3671d2e6dc05006e3b097540bfdd99ab Jul 20 13:23:20.335554 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:20.335465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c875b-ctvnl" event={"ID":"c3c826a0-10f0-4a6a-97af-44f84390b431","Type":"ContainerStarted","Data":"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71"} Jul 20 13:23:20.335554 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:20.335515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c875b-ctvnl" event={"ID":"c3c826a0-10f0-4a6a-97af-44f84390b431","Type":"ContainerStarted","Data":"ff6b18cb143b46a8ff1dc72d31277dae3671d2e6dc05006e3b097540bfdd99ab"} Jul 20 13:23:20.356960 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:20.356905 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95c875b-ctvnl" podStartSLOduration=1.356888531 podStartE2EDuration="1.356888531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:23:20.35531 +0000 UTC m=+399.021249032" watchObservedRunningTime="2026-07-20 13:23:20.356888531 +0000 UTC m=+399.022827562" Jul 20 13:23:22.371543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:22.371490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:29.908904 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:29.908867 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:29.909299 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:29.909048 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:29.913799 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:29.913774 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:30.380914 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:30.380889 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:23:30.466307 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:30.466263 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:23:44.586263 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.586196 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7784476d9f-dzgw9" podUID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" containerName="console" containerID="cri-o://4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0" gracePeriod=15 Jul 20 13:23:44.828372 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.828335 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:44.921248 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921264 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwrsk\" (UniqueName: \"kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921682 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle\") pod \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\" (UID: \"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b\") " Jul 20 13:23:44.921735 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921685 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config" (OuterVolumeSpecName: "console-config") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:44.921789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921731 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:44.921925 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca" (OuterVolumeSpecName: "service-ca") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:44.922022 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.921965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:44.923548 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.923525 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:23:44.923697 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.923666 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:23:44.923758 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:44.923716 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk" (OuterVolumeSpecName: "kube-api-access-xwrsk") pod "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" (UID: "c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b"). InnerVolumeSpecName "kube-api-access-xwrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:23:45.022470 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022433 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-oauth-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022470 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022463 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-oauth-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022470 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022475 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrsk\" (UniqueName: \"kubernetes.io/projected/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-kube-api-access-xwrsk\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022489 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-service-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022501 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022511 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-trusted-ca-bundle\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.022707 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.022522 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b-console-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:45.436787 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.436758 2575 generic.go:356] "Generic (PLEG): container finished" podID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" containerID="4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0" exitCode=0 Jul 20 13:23:45.436958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.436807 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7784476d9f-dzgw9" Jul 20 13:23:45.436958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.436852 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784476d9f-dzgw9" event={"ID":"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b","Type":"ContainerDied","Data":"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0"} Jul 20 13:23:45.436958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.436901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7784476d9f-dzgw9" event={"ID":"c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b","Type":"ContainerDied","Data":"96c79af39c709439d5fcc1ef5caabc546ea7ee9f91993356239334b15e4fabcc"} Jul 20 13:23:45.436958 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.436926 2575 scope.go:117] "RemoveContainer" containerID="4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0" Jul 20 13:23:45.445966 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.445950 2575 scope.go:117] "RemoveContainer" containerID="4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0" Jul 20 13:23:45.446219 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:23:45.446195 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0\": container with ID starting with 4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0 not found: ID does not exist" containerID="4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0" Jul 20 13:23:45.446275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.446229 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0"} err="failed to get container status \"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0\": rpc error: code = NotFound desc = could not find container \"4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0\": container with ID starting with 4a4abb6ec6eb82dbdca1a62ce178b708ffa568942efcf67c63fe34ae8bbc5ac0 not found: ID does not exist" Jul 20 13:23:45.462019 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.461990 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:45.467031 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.467006 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7784476d9f-dzgw9"] Jul 20 13:23:45.880365 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:45.880320 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" path="/var/lib/kubelet/pods/c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b/volumes" Jul 20 13:23:55.487335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.487282 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-866b57b788-bzpv8" podUID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" containerName="console" containerID="cri-o://4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911" gracePeriod=15 Jul 20 13:23:55.747023 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.746964 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:23:55.821087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh8mn\" (UniqueName: \"kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821092 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821326 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821326 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821326 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821324 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config\") pod \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\" (UID: \"cc6ebb0b-dca9-453c-85a2-9ed5bf163895\") " Jul 20 13:23:55.821741 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:55.821873 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821752 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca" (OuterVolumeSpecName: "service-ca") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:55.822027 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.821915 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config" (OuterVolumeSpecName: "console-config") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:55.822098 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.822043 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:23:55.823434 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.823404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:23:55.823434 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.823421 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:23:55.823434 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.823432 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn" (OuterVolumeSpecName: "kube-api-access-hh8mn") pod "cc6ebb0b-dca9-453c-85a2-9ed5bf163895" (UID: "cc6ebb0b-dca9-453c-85a2-9ed5bf163895"). InnerVolumeSpecName "kube-api-access-hh8mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:23:55.923003 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.922975 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh8mn\" (UniqueName: \"kubernetes.io/projected/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-kube-api-access-hh8mn\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923003 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923002 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923003 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923012 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-oauth-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923020 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-trusted-ca-bundle\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923030 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-service-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923039 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-oauth-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:55.923228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:55.923056 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc6ebb0b-dca9-453c-85a2-9ed5bf163895-console-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:23:56.479959 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.479929 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" containerID="4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911" exitCode=0 Jul 20 13:23:56.480131 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.479973 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866b57b788-bzpv8" Jul 20 13:23:56.480131 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.479998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b57b788-bzpv8" event={"ID":"cc6ebb0b-dca9-453c-85a2-9ed5bf163895","Type":"ContainerDied","Data":"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911"} Jul 20 13:23:56.480131 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.480038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866b57b788-bzpv8" event={"ID":"cc6ebb0b-dca9-453c-85a2-9ed5bf163895","Type":"ContainerDied","Data":"9c4c240ae609c60cd08e6a3d3b9232abdd0524a1a6fe7f22a931b95053ff45b0"} Jul 20 13:23:56.480131 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.480059 2575 scope.go:117] "RemoveContainer" containerID="4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911" Jul 20 13:23:56.488317 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.488115 2575 scope.go:117] "RemoveContainer" containerID="4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911" Jul 20 13:23:56.488662 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:23:56.488409 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911\": container with ID starting with 4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911 not found: ID does not exist" containerID="4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911" Jul 20 13:23:56.488662 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.488437 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911"} err="failed to get container status \"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911\": rpc error: code = NotFound desc = could not find container \"4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911\": container with ID starting with 4546c3f0d96cf4a3c5cf775db0508a8f1fb3518187c8e548ca0c34b633074911 not found: ID does not exist" Jul 20 13:23:56.499484 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.499464 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:23:56.505199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:56.505177 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-866b57b788-bzpv8"] Jul 20 13:23:57.880705 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:23:57.880672 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" path="/var/lib/kubelet/pods/cc6ebb0b-dca9-453c-85a2-9ed5bf163895/volumes" Jul 20 13:25:06.877025 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.876970 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7hpb"] Jul 20 13:25:06.877683 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877661 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" containerName="console" Jul 20 13:25:06.877769 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877686 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" containerName="console" Jul 20 13:25:06.877769 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877733 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" containerName="console" Jul 20 13:25:06.877769 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877741 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" containerName="console" Jul 20 13:25:06.877905 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877882 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6ebb0b-dca9-453c-85a2-9ed5bf163895" containerName="console" Jul 20 13:25:06.877905 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.877899 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45b02f9-fb4c-4d6e-a9e0-86e933c6f81b" containerName="console" Jul 20 13:25:06.880500 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.880476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:06.887427 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.887406 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 13:25:06.887537 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.887483 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bs89q\"" Jul 20 13:25:06.890789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.890766 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 13:25:06.893690 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.893666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7hpb"] Jul 20 13:25:06.999835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.999801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.000017 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:06.999855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6drd\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-kube-api-access-d6drd\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.100232 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.100200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.100441 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.100274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6drd\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-kube-api-access-d6drd\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.109850 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.109829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.110185 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.110167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6drd\" (UniqueName: \"kubernetes.io/projected/8c5caba4-09d1-41c3-b96a-ff1505f562f4-kube-api-access-d6drd\") pod \"cert-manager-webhook-6486ffd997-m7hpb\" (UID: \"8c5caba4-09d1-41c3-b96a-ff1505f562f4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.190420 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.190374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:07.326094 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.326066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7hpb"] Jul 20 13:25:07.328142 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:25:07.328112 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c5caba4_09d1_41c3_b96a_ff1505f562f4.slice/crio-f29d079182cfd50671cd6ff353b5d3066c699bca49f9903ea63f71591683aaa0 WatchSource:0}: Error finding container f29d079182cfd50671cd6ff353b5d3066c699bca49f9903ea63f71591683aaa0: Status 404 returned error can't find the container with id f29d079182cfd50671cd6ff353b5d3066c699bca49f9903ea63f71591683aaa0 Jul 20 13:25:07.770741 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:07.770709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" event={"ID":"8c5caba4-09d1-41c3-b96a-ff1505f562f4","Type":"ContainerStarted","Data":"f29d079182cfd50671cd6ff353b5d3066c699bca49f9903ea63f71591683aaa0"} Jul 20 13:25:10.785673 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:10.785641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" event={"ID":"8c5caba4-09d1-41c3-b96a-ff1505f562f4","Type":"ContainerStarted","Data":"58b19f70e6942e054c5433db06bea0e7ea4c085a7ab98bc7a124295bb1ca04cf"} Jul 20 13:25:10.786080 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:10.785693 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:10.806081 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:10.806030 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" podStartSLOduration=1.985414909 podStartE2EDuration="4.806011206s" totalImagesPullingTime="2.820596297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:10.804032668 +0000 UTC m=+509.469971699" watchObservedRunningTime="2026-07-20 13:25:10.806011206 +0000 UTC m=+509.471950236" Jul 20 13:25:13.306893 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.306856 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c"] Jul 20 13:25:13.310071 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.310055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.312885 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.312860 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:25:13.312996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.312861 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xvcvq\"" Jul 20 13:25:13.313974 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.313960 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 13:25:13.322893 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.322867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c"] Jul 20 13:25:13.462424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.462350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-kube-api-access-9sn4d\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.462424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.462430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.563851 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.563759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-kube-api-access-9sn4d\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.563851 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.563797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.564244 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.564224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.572105 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.572067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91-kube-api-access-9sn4d\") pod \"openshift-lws-operator-fd8ccff4c-wjm9c\" (UID: \"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.618568 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.618537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" Jul 20 13:25:13.750998 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.750965 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c"] Jul 20 13:25:13.754155 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:25:13.754130 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee9ba7ae_4763_40b9_b0d5_4ea2deb6ef91.slice/crio-81258f3e37cd738a39ee00b98a698d1396a73b36b1a2f06d98f937d299682edf WatchSource:0}: Error finding container 81258f3e37cd738a39ee00b98a698d1396a73b36b1a2f06d98f937d299682edf: Status 404 returned error can't find the container with id 81258f3e37cd738a39ee00b98a698d1396a73b36b1a2f06d98f937d299682edf Jul 20 13:25:13.798133 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:13.798099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" event={"ID":"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91","Type":"ContainerStarted","Data":"81258f3e37cd738a39ee00b98a698d1396a73b36b1a2f06d98f937d299682edf"} Jul 20 13:25:16.792264 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:16.792231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m7hpb" Jul 20 13:25:16.814788 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:16.814750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" event={"ID":"ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91","Type":"ContainerStarted","Data":"3754483a1d9c0012c66d6dd79ef76701492f5019b0d416731ec9c51f2864ca1d"} Jul 20 13:25:16.840370 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:16.840308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wjm9c" podStartSLOduration=1.503330782 podStartE2EDuration="3.840291725s" totalImagesPullingTime="2.336960943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:16.837874632 +0000 UTC m=+515.503813665" watchObservedRunningTime="2026-07-20 13:25:16.840291725 +0000 UTC m=+515.506230755" Jul 20 13:25:35.357466 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.357382 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp"] Jul 20 13:25:35.386976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.386945 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp"] Jul 20 13:25:35.387138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.387046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.389555 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.389531 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ggvq6\"" Jul 20 13:25:35.389903 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.389884 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 13:25:35.389986 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.389939 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 13:25:35.390206 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.390190 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 13:25:35.390395 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.390382 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 13:25:35.450227 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.450191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.450456 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.450252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.450456 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.450340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.450456 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.450426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.450599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.450458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pff7\" (UniqueName: \"kubernetes.io/projected/4425b323-c82f-4897-a79f-9410096fba6a-kube-api-access-2pff7\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551322 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551547 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pff7\" (UniqueName: \"kubernetes.io/projected/4425b323-c82f-4897-a79f-9410096fba6a-kube-api-access-2pff7\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551547 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551547 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551718 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.551898 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.551879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.553724 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.553702 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4425b323-c82f-4897-a79f-9410096fba6a-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.553930 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.553913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.553982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.553918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4425b323-c82f-4897-a79f-9410096fba6a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.562111 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.562090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pff7\" (UniqueName: \"kubernetes.io/projected/4425b323-c82f-4897-a79f-9410096fba6a-kube-api-access-2pff7\") pod \"opendatahub-operator-controller-manager-864d7fc49d-hw7xp\" (UID: \"4425b323-c82f-4897-a79f-9410096fba6a\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.698019 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.697971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:35.867258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.867233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp"] Jul 20 13:25:35.870069 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:25:35.870043 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4425b323_c82f_4897_a79f_9410096fba6a.slice/crio-32d57753cae0085834996a04626d235c1ff0a3ac4144c672ac6dc207f49a8423 WatchSource:0}: Error finding container 32d57753cae0085834996a04626d235c1ff0a3ac4144c672ac6dc207f49a8423: Status 404 returned error can't find the container with id 32d57753cae0085834996a04626d235c1ff0a3ac4144c672ac6dc207f49a8423 Jul 20 13:25:35.890772 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:35.890739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" event={"ID":"4425b323-c82f-4897-a79f-9410096fba6a","Type":"ContainerStarted","Data":"32d57753cae0085834996a04626d235c1ff0a3ac4144c672ac6dc207f49a8423"} Jul 20 13:25:38.907249 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:38.907217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" event={"ID":"4425b323-c82f-4897-a79f-9410096fba6a","Type":"ContainerStarted","Data":"df6d5b06e494bbe290653540b73a2192fa55cd566780b16c9856cd82c7e5bea2"} Jul 20 13:25:40.917836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:40.917798 2575 generic.go:356] "Generic (PLEG): container finished" podID="4425b323-c82f-4897-a79f-9410096fba6a" containerID="df6d5b06e494bbe290653540b73a2192fa55cd566780b16c9856cd82c7e5bea2" exitCode=0 Jul 20 13:25:40.918231 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:40.917872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" event={"ID":"4425b323-c82f-4897-a79f-9410096fba6a","Type":"ContainerDied","Data":"df6d5b06e494bbe290653540b73a2192fa55cd566780b16c9856cd82c7e5bea2"} Jul 20 13:25:41.923738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:41.923696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" event={"ID":"4425b323-c82f-4897-a79f-9410096fba6a","Type":"ContainerStarted","Data":"67c5fc0f1e9ab9741b2311a8f37850e11feb6ffb59f1da2d8dc71049b9c3356d"} Jul 20 13:25:41.924181 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:41.924016 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:41.954917 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:41.954868 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" podStartSLOduration=2.344559083 podStartE2EDuration="6.954851267s" totalImagesPullingTime="3.452837084s" totalInitContainerRuntime="1.1574551s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:41.953210866 +0000 UTC m=+540.619149895" watchObservedRunningTime="2026-07-20 13:25:41.954851267 +0000 UTC m=+540.620790297" Jul 20 13:25:52.930175 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:52.930147 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-hw7xp" Jul 20 13:25:55.747490 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.747449 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl"] Jul 20 13:25:55.753827 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.753800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.757795 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.757765 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pmk9l\"" Jul 20 13:25:55.759206 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.759184 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 13:25:55.759348 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.759234 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 13:25:55.769563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.769536 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl"] Jul 20 13:25:55.829030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.828992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2p5z\" (UniqueName: \"kubernetes.io/projected/20b277b7-d465-488a-bf2a-622498290e40-kube-api-access-f2p5z\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.829245 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.829114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20b277b7-d465-488a-bf2a-622498290e40-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.829245 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.829169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b277b7-d465-488a-bf2a-622498290e40-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.930097 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.930059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20b277b7-d465-488a-bf2a-622498290e40-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.930267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.930146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b277b7-d465-488a-bf2a-622498290e40-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.930267 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.930188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2p5z\" (UniqueName: \"kubernetes.io/projected/20b277b7-d465-488a-bf2a-622498290e40-kube-api-access-f2p5z\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.932503 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.932478 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20b277b7-d465-488a-bf2a-622498290e40-tmp\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.932810 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.932788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b277b7-d465-488a-bf2a-622498290e40-tls-certs\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:55.946203 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:55.946171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2p5z\" (UniqueName: \"kubernetes.io/projected/20b277b7-d465-488a-bf2a-622498290e40-kube-api-access-f2p5z\") pod \"kube-auth-proxy-5c5c8fcf86-8w6nl\" (UID: \"20b277b7-d465-488a-bf2a-622498290e40\") " pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:56.065087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:56.065004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" Jul 20 13:25:56.210427 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:56.210397 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl"] Jul 20 13:25:56.213830 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:25:56.213793 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b277b7_d465_488a_bf2a_622498290e40.slice/crio-43b9c2aff66964b42dd49cc5acbd486db1e6051acd5bdcc2d34fb819e4599ffd WatchSource:0}: Error finding container 43b9c2aff66964b42dd49cc5acbd486db1e6051acd5bdcc2d34fb819e4599ffd: Status 404 returned error can't find the container with id 43b9c2aff66964b42dd49cc5acbd486db1e6051acd5bdcc2d34fb819e4599ffd Jul 20 13:25:56.999491 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:56.999457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" event={"ID":"20b277b7-d465-488a-bf2a-622498290e40","Type":"ContainerStarted","Data":"43b9c2aff66964b42dd49cc5acbd486db1e6051acd5bdcc2d34fb819e4599ffd"} Jul 20 13:25:57.035590 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.035551 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdnv"] Jul 20 13:25:57.038228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.038204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.040941 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.040921 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 20 13:25:57.041086 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.041064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rdfvn\"" Jul 20 13:25:57.057399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.057371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdnv"] Jul 20 13:25:57.141760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.141724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8cw\" (UniqueName: \"kubernetes.io/projected/9ea63735-8420-4324-8fc7-d90dea2bf75c-kube-api-access-tv8cw\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.141948 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.141775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea63735-8420-4324-8fc7-d90dea2bf75c-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.242570 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.242527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8cw\" (UniqueName: \"kubernetes.io/projected/9ea63735-8420-4324-8fc7-d90dea2bf75c-kube-api-access-tv8cw\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.242759 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.242589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea63735-8420-4324-8fc7-d90dea2bf75c-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.245539 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.245510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea63735-8420-4324-8fc7-d90dea2bf75c-tls-certs\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.254784 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.254708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8cw\" (UniqueName: \"kubernetes.io/projected/9ea63735-8420-4324-8fc7-d90dea2bf75c-kube-api-access-tv8cw\") pod \"model-serving-api-54d89bd5fc-nkdnv\" (UID: \"9ea63735-8420-4324-8fc7-d90dea2bf75c\") " pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.351933 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.351895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:25:57.501141 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:57.501116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nkdnv"] Jul 20 13:25:57.503832 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:25:57.503807 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea63735_8420_4324_8fc7_d90dea2bf75c.slice/crio-e9a9948b79840c47d3631a2a76a1ae1229f39c82e4fb0e0670ccaabfaa8a41da WatchSource:0}: Error finding container e9a9948b79840c47d3631a2a76a1ae1229f39c82e4fb0e0670ccaabfaa8a41da: Status 404 returned error can't find the container with id e9a9948b79840c47d3631a2a76a1ae1229f39c82e4fb0e0670ccaabfaa8a41da Jul 20 13:25:58.006080 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:25:58.006043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" event={"ID":"9ea63735-8420-4324-8fc7-d90dea2bf75c","Type":"ContainerStarted","Data":"e9a9948b79840c47d3631a2a76a1ae1229f39c82e4fb0e0670ccaabfaa8a41da"} Jul 20 13:26:00.017450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:00.017417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" event={"ID":"20b277b7-d465-488a-bf2a-622498290e40","Type":"ContainerStarted","Data":"61e0313e3d7c13128c05c7e2d190c29c40751acdf6e0f8da131c432c076d4a9f"} Jul 20 13:26:00.018985 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:00.018956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" event={"ID":"9ea63735-8420-4324-8fc7-d90dea2bf75c","Type":"ContainerStarted","Data":"2f9bb590752bc47688cc913920f3e6d47c2c13121988b054804a8e7e6f81f98d"} Jul 20 13:26:00.019132 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:00.019013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:26:00.043798 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:00.043749 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c5c8fcf86-8w6nl" podStartSLOduration=1.990237247 podStartE2EDuration="5.043733889s" totalImagesPullingTime="3.053496642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:00.042018273 +0000 UTC m=+558.707957306" watchObservedRunningTime="2026-07-20 13:26:00.043733889 +0000 UTC m=+558.709672919" Jul 20 13:26:02.784349 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.784294 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" podStartSLOduration=5.022884887 podStartE2EDuration="6.784272604s" totalImagesPullingTime="1.761387717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:00.071511441 +0000 UTC m=+558.737450473" watchObservedRunningTime="2026-07-20 13:26:02.784272604 +0000 UTC m=+561.450211650" Jul 20 13:26:02.784817 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.784800 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9"] Jul 20 13:26:02.787629 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.787613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.791570 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.791546 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 13:26:02.791699 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.791657 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gqj4c\"" Jul 20 13:26:02.806467 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.806437 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9"] Jul 20 13:26:02.902453 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902453 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8xz\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-kube-api-access-gf8xz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:02.902681 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:02.902668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.003939 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.003893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004149 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.003993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004293 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8xz\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-kube-api-access-gf8xz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004525 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004525 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004638 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004638 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004744 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.004892 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.004870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.005202 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.005185 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.006472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.006446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.006688 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.006672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.023967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.023940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8xz\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-kube-api-access-gf8xz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.025492 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.025465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm5g9\" (UID: \"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.098004 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.097916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:03.252920 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:03.252880 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9"] Jul 20 13:26:03.255842 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:26:03.255815 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c84e28a_d2b9_4aaf_aa4c_1886aea1d0af.slice/crio-fb01c56009526c5b8b0a0bed0a8a11a5bbc2dfdfbe4c79999841cf985916ccdf WatchSource:0}: Error finding container fb01c56009526c5b8b0a0bed0a8a11a5bbc2dfdfbe4c79999841cf985916ccdf: Status 404 returned error can't find the container with id fb01c56009526c5b8b0a0bed0a8a11a5bbc2dfdfbe4c79999841cf985916ccdf Jul 20 13:26:04.040878 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:04.040841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" event={"ID":"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af","Type":"ContainerStarted","Data":"fb01c56009526c5b8b0a0bed0a8a11a5bbc2dfdfbe4c79999841cf985916ccdf"} Jul 20 13:26:05.710741 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.710706 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-cn7cm"] Jul 20 13:26:05.717844 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.717816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.722487 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.722461 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 13:26:05.723665 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.723645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fwswp\"" Jul 20 13:26:05.723808 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.723664 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 13:26:05.724051 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.724030 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 13:26:05.731538 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.731513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-cn7cm"] Jul 20 13:26:05.832485 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.832450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-manager-config\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.832485 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.832492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.832756 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.832524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-metrics-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.832756 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.832609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwvj\" (UniqueName: \"kubernetes.io/projected/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-kube-api-access-6rwvj\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.933763 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.933715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-manager-config\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.933959 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.933775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.933959 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.933807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-metrics-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.933959 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.933865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwvj\" (UniqueName: \"kubernetes.io/projected/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-kube-api-access-6rwvj\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.934569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.934542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-manager-config\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.936671 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.936643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.936802 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.936732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-metrics-cert\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:05.947805 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:05.947780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwvj\" (UniqueName: \"kubernetes.io/projected/c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f-kube-api-access-6rwvj\") pod \"lws-controller-manager-548487689c-cn7cm\" (UID: \"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:06.032194 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:06.032111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:06.099476 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:06.099426 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:26:06.099591 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:06.099534 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:26:06.099591 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:06.099582 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:26:06.208600 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:06.208568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-cn7cm"] Jul 20 13:26:06.210602 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:26:06.210570 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c0ca60_20ae_4f3e_a9ef_f577f8733a0f.slice/crio-1a46d3d6fcf784e17a077ec021e167d10df9efb7be6ae1feaf0e23d80b585116 WatchSource:0}: Error finding container 1a46d3d6fcf784e17a077ec021e167d10df9efb7be6ae1feaf0e23d80b585116: Status 404 returned error can't find the container with id 1a46d3d6fcf784e17a077ec021e167d10df9efb7be6ae1feaf0e23d80b585116 Jul 20 13:26:07.063819 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:07.063779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" event={"ID":"7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af","Type":"ContainerStarted","Data":"b2e792bc8367812843204444d2ccc2e0e5578e46152d1457e9177df0c74bbebe"} Jul 20 13:26:07.066000 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:07.065958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" event={"ID":"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f","Type":"ContainerStarted","Data":"1a46d3d6fcf784e17a077ec021e167d10df9efb7be6ae1feaf0e23d80b585116"} Jul 20 13:26:07.098537 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:07.098488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:07.104368 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:07.104319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:07.113540 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:07.113475 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" podStartSLOduration=2.272317054 podStartE2EDuration="5.11345246s" totalImagesPullingTime="2.841135406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:07.112413689 +0000 UTC m=+565.778352722" watchObservedRunningTime="2026-07-20 13:26:07.11345246 +0000 UTC m=+565.779391492" Jul 20 13:26:08.072123 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:08.072028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" event={"ID":"c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f","Type":"ContainerStarted","Data":"5ce0663d2b05860ec8626afb507a34484cd534f59b85d14308e4ae94323800d0"} Jul 20 13:26:08.072123 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:08.072110 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:26:08.072599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:08.072338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:08.073282 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:08.073266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm5g9" Jul 20 13:26:08.102993 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:08.102946 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" podStartSLOduration=1.613891439 podStartE2EDuration="3.102924694s" totalImagesPullingTime="1.489033255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:08.102735454 +0000 UTC m=+566.768674485" watchObservedRunningTime="2026-07-20 13:26:08.102924694 +0000 UTC m=+566.768863724" Jul 20 13:26:11.027546 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:11.027514 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nkdnv" Jul 20 13:26:19.078435 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:26:19.078391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-548487689c-cn7cm" Jul 20 13:28:16.629052 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.629017 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f6d9b5758-54mgr"] Jul 20 13:28:16.632477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.632450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.645679 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.645653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6d9b5758-54mgr"] Jul 20 13:28:16.704713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-oauth-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.704713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.705000 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.705000 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-trusted-ca-bundle\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.705000 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-service-ca\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.705000 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.704964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-oauth-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.705157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.705021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsrv\" (UniqueName: \"kubernetes.io/projected/45aabc49-e711-4d00-878d-0c13f7bfaf60-kube-api-access-xgsrv\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.805855 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.805820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806048 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.805865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806048 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.805959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-trusted-ca-bundle\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806048 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.805995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-service-ca\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806048 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-oauth-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsrv\" (UniqueName: \"kubernetes.io/projected/45aabc49-e711-4d00-878d-0c13f7bfaf60-kube-api-access-xgsrv\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-oauth-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806697 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806671 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806824 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-service-ca\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806824 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-oauth-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.806954 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.806938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45aabc49-e711-4d00-878d-0c13f7bfaf60-trusted-ca-bundle\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.808569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.808536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-oauth-config\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.808735 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.808717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45aabc49-e711-4d00-878d-0c13f7bfaf60-console-serving-cert\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.819735 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.819710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsrv\" (UniqueName: \"kubernetes.io/projected/45aabc49-e711-4d00-878d-0c13f7bfaf60-kube-api-access-xgsrv\") pod \"console-6f6d9b5758-54mgr\" (UID: \"45aabc49-e711-4d00-878d-0c13f7bfaf60\") " pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:16.942543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:16.942438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:17.089028 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:17.088840 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6d9b5758-54mgr"] Jul 20 13:28:17.091875 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:28:17.091846 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45aabc49_e711_4d00_878d_0c13f7bfaf60.slice/crio-473d41c4590ec7611115975b6a3ac56de50e7ee57045b685b3b5da8a03dd6f74 WatchSource:0}: Error finding container 473d41c4590ec7611115975b6a3ac56de50e7ee57045b685b3b5da8a03dd6f74: Status 404 returned error can't find the container with id 473d41c4590ec7611115975b6a3ac56de50e7ee57045b685b3b5da8a03dd6f74 Jul 20 13:28:17.653521 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:17.653487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6d9b5758-54mgr" event={"ID":"45aabc49-e711-4d00-878d-0c13f7bfaf60","Type":"ContainerStarted","Data":"1295ba1f22f22bc67bbc09d93da206bf39a76ea0277e3143fcdf3fff65c9cd88"} Jul 20 13:28:17.654014 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:17.653545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6d9b5758-54mgr" event={"ID":"45aabc49-e711-4d00-878d-0c13f7bfaf60","Type":"ContainerStarted","Data":"473d41c4590ec7611115975b6a3ac56de50e7ee57045b685b3b5da8a03dd6f74"} Jul 20 13:28:17.681858 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:17.681807 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f6d9b5758-54mgr" podStartSLOduration=1.681788669 podStartE2EDuration="1.681788669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:17.680115878 +0000 UTC m=+696.346054898" watchObservedRunningTime="2026-07-20 13:28:17.681788669 +0000 UTC m=+696.347727698" Jul 20 13:28:26.942900 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:26.942842 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:26.943412 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:26.942921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:26.947744 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:26.947719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:27.705170 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:27.705143 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f6d9b5758-54mgr" Jul 20 13:28:27.748106 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:27.748073 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:28:42.203630 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.203554 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:42.207091 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.207070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:42.209814 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.209792 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 13:28:42.209939 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.209907 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 13:28:42.210976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.210958 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-94vrh\"" Jul 20 13:28:42.213762 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.213740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:42.231689 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.231652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8xl\" (UniqueName: \"kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl\") pod \"authorino-58cd955675-tbn5r\" (UID: \"4aa2e897-41a5-44ce-9205-34380cb6552e\") " pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:42.332523 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.332486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8xl\" (UniqueName: \"kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl\") pod \"authorino-58cd955675-tbn5r\" (UID: \"4aa2e897-41a5-44ce-9205-34380cb6552e\") " pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:42.342565 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.342536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8xl\" (UniqueName: \"kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl\") pod \"authorino-58cd955675-tbn5r\" (UID: \"4aa2e897-41a5-44ce-9205-34380cb6552e\") " pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:42.517280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.517194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:42.525338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.525316 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:28:42.529521 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.529500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:28:42.535986 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.535960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:28:42.635129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.635090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84js7\" (UniqueName: \"kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7\") pod \"authorino-75d556bd77-nrxjg\" (UID: \"da02579f-2d75-4aff-bc51-2cdcd9042ba8\") " pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:28:42.653199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.653170 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:42.656448 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:28:42.656411 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4aa2e897_41a5_44ce_9205_34380cb6552e.slice/crio-9e581390c161859cda66ebec1cd3aff73daab847568f07be471b460987c5e6fc WatchSource:0}: Error finding container 9e581390c161859cda66ebec1cd3aff73daab847568f07be471b460987c5e6fc: Status 404 returned error can't find the container with id 9e581390c161859cda66ebec1cd3aff73daab847568f07be471b460987c5e6fc Jul 20 13:28:42.657835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.657821 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:28:42.736002 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.735961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84js7\" (UniqueName: \"kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7\") pod \"authorino-75d556bd77-nrxjg\" (UID: \"da02579f-2d75-4aff-bc51-2cdcd9042ba8\") " pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:28:42.747216 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.747181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84js7\" (UniqueName: \"kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7\") pod \"authorino-75d556bd77-nrxjg\" (UID: \"da02579f-2d75-4aff-bc51-2cdcd9042ba8\") " pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:28:42.770855 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.770768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbn5r" event={"ID":"4aa2e897-41a5-44ce-9205-34380cb6552e","Type":"ContainerStarted","Data":"9e581390c161859cda66ebec1cd3aff73daab847568f07be471b460987c5e6fc"} Jul 20 13:28:42.853284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.853243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:28:42.985478 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:42.985453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:28:42.987585 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:28:42.987564 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda02579f_2d75_4aff_bc51_2cdcd9042ba8.slice/crio-3724ec8bbdbc1eb3f57f3af7b35cf599308ddd1b251fb044c24a3cf867a9e03a WatchSource:0}: Error finding container 3724ec8bbdbc1eb3f57f3af7b35cf599308ddd1b251fb044c24a3cf867a9e03a: Status 404 returned error can't find the container with id 3724ec8bbdbc1eb3f57f3af7b35cf599308ddd1b251fb044c24a3cf867a9e03a Jul 20 13:28:43.782778 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:43.782700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nrxjg" event={"ID":"da02579f-2d75-4aff-bc51-2cdcd9042ba8","Type":"ContainerStarted","Data":"3724ec8bbdbc1eb3f57f3af7b35cf599308ddd1b251fb044c24a3cf867a9e03a"} Jul 20 13:28:46.800506 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:46.800414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nrxjg" event={"ID":"da02579f-2d75-4aff-bc51-2cdcd9042ba8","Type":"ContainerStarted","Data":"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472"} Jul 20 13:28:46.801819 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:46.801796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbn5r" event={"ID":"4aa2e897-41a5-44ce-9205-34380cb6552e","Type":"ContainerStarted","Data":"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db"} Jul 20 13:28:46.832665 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:46.832614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nrxjg" podStartSLOduration=1.4338622380000001 podStartE2EDuration="4.832594374s" totalImagesPullingTime="3.398732136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:46.832243445 +0000 UTC m=+725.498182489" watchObservedRunningTime="2026-07-20 13:28:46.832594374 +0000 UTC m=+725.498533407" Jul 20 13:28:46.867187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:46.867135 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tbn5r" podStartSLOduration=1.123693875 podStartE2EDuration="4.86711758s" totalImagesPullingTime="3.743423705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:46.866752611 +0000 UTC m=+725.532691645" watchObservedRunningTime="2026-07-20 13:28:46.86711758 +0000 UTC m=+725.533056611" Jul 20 13:28:46.912698 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:46.912659 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:48.811592 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:48.811534 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tbn5r" podUID="4aa2e897-41a5-44ce-9205-34380cb6552e" containerName="authorino" containerID="cri-o://bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db" gracePeriod=30 Jul 20 13:28:49.069891 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.069836 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:49.201500 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.201469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8xl\" (UniqueName: \"kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl\") pod \"4aa2e897-41a5-44ce-9205-34380cb6552e\" (UID: \"4aa2e897-41a5-44ce-9205-34380cb6552e\") " Jul 20 13:28:49.203626 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.203591 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl" (OuterVolumeSpecName: "kube-api-access-5t8xl") pod "4aa2e897-41a5-44ce-9205-34380cb6552e" (UID: "4aa2e897-41a5-44ce-9205-34380cb6552e"). InnerVolumeSpecName "kube-api-access-5t8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:49.302556 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.302505 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8xl\" (UniqueName: \"kubernetes.io/projected/4aa2e897-41a5-44ce-9205-34380cb6552e-kube-api-access-5t8xl\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:49.817734 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.817699 2575 generic.go:356] "Generic (PLEG): container finished" podID="4aa2e897-41a5-44ce-9205-34380cb6552e" containerID="bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db" exitCode=0 Jul 20 13:28:49.818191 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.817745 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbn5r" Jul 20 13:28:49.818191 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.817747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbn5r" event={"ID":"4aa2e897-41a5-44ce-9205-34380cb6552e","Type":"ContainerDied","Data":"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db"} Jul 20 13:28:49.818191 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.817783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbn5r" event={"ID":"4aa2e897-41a5-44ce-9205-34380cb6552e","Type":"ContainerDied","Data":"9e581390c161859cda66ebec1cd3aff73daab847568f07be471b460987c5e6fc"} Jul 20 13:28:49.818191 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.817803 2575 scope.go:117] "RemoveContainer" containerID="bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db" Jul 20 13:28:49.826764 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.826742 2575 scope.go:117] "RemoveContainer" containerID="bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db" Jul 20 13:28:49.827026 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:28:49.827009 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db\": container with ID starting with bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db not found: ID does not exist" containerID="bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db" Jul 20 13:28:49.827090 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.827035 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db"} err="failed to get container status \"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db\": rpc error: code = NotFound desc = could not find container \"bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db\": container with ID starting with bf54c63a5ab061c2ead0342502f77f213a7178a9a50ef8473aa613cab7fad2db not found: ID does not exist" Jul 20 13:28:49.838616 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.838591 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:49.842557 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.842537 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbn5r"] Jul 20 13:28:49.881466 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:49.881440 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa2e897-41a5-44ce-9205-34380cb6552e" path="/var/lib/kubelet/pods/4aa2e897-41a5-44ce-9205-34380cb6552e/volumes" Jul 20 13:28:52.771721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:52.771675 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95c875b-ctvnl" podUID="c3c826a0-10f0-4a6a-97af-44f84390b431" containerName="console" containerID="cri-o://07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71" gracePeriod=15 Jul 20 13:28:53.023253 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.023184 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:28:53.134924 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.134888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.134924 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.134933 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135125 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.134980 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135125 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135084 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135188 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135156 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135188 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135181 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7s92\" (UniqueName: \"kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135333 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135216 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config\") pod \"c3c826a0-10f0-4a6a-97af-44f84390b431\" (UID: \"c3c826a0-10f0-4a6a-97af-44f84390b431\") " Jul 20 13:28:53.135463 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135349 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config" (OuterVolumeSpecName: "console-config") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:28:53.135463 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135389 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:28:53.135570 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135558 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-console-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.135623 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135571 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-service-ca\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.135623 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135579 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:28:53.135772 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.135754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:28:53.137349 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.137324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:28:53.137778 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.137759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:28:53.137852 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.137772 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92" (OuterVolumeSpecName: "kube-api-access-v7s92") pod "c3c826a0-10f0-4a6a-97af-44f84390b431" (UID: "c3c826a0-10f0-4a6a-97af-44f84390b431"). InnerVolumeSpecName "kube-api-access-v7s92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:28:53.236818 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.236779 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-oauth-config\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.236818 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.236811 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-trusted-ca-bundle\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.236818 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.236820 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3c826a0-10f0-4a6a-97af-44f84390b431-console-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.236818 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.236829 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3c826a0-10f0-4a6a-97af-44f84390b431-oauth-serving-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.237074 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.236837 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7s92\" (UniqueName: \"kubernetes.io/projected/c3c826a0-10f0-4a6a-97af-44f84390b431-kube-api-access-v7s92\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:28:53.837989 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.837954 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3c826a0-10f0-4a6a-97af-44f84390b431" containerID="07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71" exitCode=0 Jul 20 13:28:53.838477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.838009 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c875b-ctvnl" Jul 20 13:28:53.838477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.838006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c875b-ctvnl" event={"ID":"c3c826a0-10f0-4a6a-97af-44f84390b431","Type":"ContainerDied","Data":"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71"} Jul 20 13:28:53.838477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.838151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c875b-ctvnl" event={"ID":"c3c826a0-10f0-4a6a-97af-44f84390b431","Type":"ContainerDied","Data":"ff6b18cb143b46a8ff1dc72d31277dae3671d2e6dc05006e3b097540bfdd99ab"} Jul 20 13:28:53.838477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.838190 2575 scope.go:117] "RemoveContainer" containerID="07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71" Jul 20 13:28:53.847563 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.847547 2575 scope.go:117] "RemoveContainer" containerID="07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71" Jul 20 13:28:53.847827 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:28:53.847808 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71\": container with ID starting with 07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71 not found: ID does not exist" containerID="07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71" Jul 20 13:28:53.847876 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.847836 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71"} err="failed to get container status \"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71\": rpc error: code = NotFound desc = could not find container \"07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71\": container with ID starting with 07654c71e76db7f8fca8aa5d668cef99996a5d009e6ad70305b238399fa38d71 not found: ID does not exist" Jul 20 13:28:53.859783 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.859761 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:28:53.865601 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.865578 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95c875b-ctvnl"] Jul 20 13:28:53.881631 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:28:53.881605 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c826a0-10f0-4a6a-97af-44f84390b431" path="/var/lib/kubelet/pods/c3c826a0-10f0-4a6a-97af-44f84390b431/volumes" Jul 20 13:29:11.100698 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.100657 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:11.101182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101167 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c826a0-10f0-4a6a-97af-44f84390b431" containerName="console" Jul 20 13:29:11.101227 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101185 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c826a0-10f0-4a6a-97af-44f84390b431" containerName="console" Jul 20 13:29:11.101259 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101225 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa2e897-41a5-44ce-9205-34380cb6552e" containerName="authorino" Jul 20 13:29:11.101259 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101234 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa2e897-41a5-44ce-9205-34380cb6552e" containerName="authorino" Jul 20 13:29:11.101317 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101307 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c826a0-10f0-4a6a-97af-44f84390b431" containerName="console" Jul 20 13:29:11.101369 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.101321 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa2e897-41a5-44ce-9205-34380cb6552e" containerName="authorino" Jul 20 13:29:11.104576 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.104558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:11.108530 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.108502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:11.210760 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.210718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzbqj\" (UniqueName: \"kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj\") pod \"authorino-5d4b67c446-jscr5\" (UID: \"19c90571-a3ac-4d34-b370-a31024eb9636\") " pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:11.312297 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.312239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzbqj\" (UniqueName: \"kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj\") pod \"authorino-5d4b67c446-jscr5\" (UID: \"19c90571-a3ac-4d34-b370-a31024eb9636\") " pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:11.320281 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.320250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzbqj\" (UniqueName: \"kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj\") pod \"authorino-5d4b67c446-jscr5\" (UID: \"19c90571-a3ac-4d34-b370-a31024eb9636\") " pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:11.344528 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.344493 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:11.344827 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.344811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:11.486674 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.486640 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:11.490570 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:29:11.490542 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19c90571_a3ac_4d34_b370_a31024eb9636.slice/crio-34ded4aafe33f81c6019fc57b3bf4761677a8b57631d6d5c951f2106489c17c9 WatchSource:0}: Error finding container 34ded4aafe33f81c6019fc57b3bf4761677a8b57631d6d5c951f2106489c17c9: Status 404 returned error can't find the container with id 34ded4aafe33f81c6019fc57b3bf4761677a8b57631d6d5c951f2106489c17c9 Jul 20 13:29:11.930551 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:11.930512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jscr5" event={"ID":"19c90571-a3ac-4d34-b370-a31024eb9636","Type":"ContainerStarted","Data":"34ded4aafe33f81c6019fc57b3bf4761677a8b57631d6d5c951f2106489c17c9"} Jul 20 13:29:12.936141 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:12.936092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jscr5" event={"ID":"19c90571-a3ac-4d34-b370-a31024eb9636","Type":"ContainerStarted","Data":"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325"} Jul 20 13:29:12.936587 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:12.936175 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jscr5" podUID="19c90571-a3ac-4d34-b370-a31024eb9636" containerName="authorino" containerID="cri-o://4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325" gracePeriod=30 Jul 20 13:29:12.952763 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:12.952719 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jscr5" podStartSLOduration=1.049036122 podStartE2EDuration="1.952702963s" totalImagesPullingTime="903.666841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:29:12.951051518 +0000 UTC m=+751.616990549" watchObservedRunningTime="2026-07-20 13:29:12.952702963 +0000 UTC m=+751.618641994" Jul 20 13:29:13.192450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.192387 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:13.329296 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.329251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzbqj\" (UniqueName: \"kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj\") pod \"19c90571-a3ac-4d34-b370-a31024eb9636\" (UID: \"19c90571-a3ac-4d34-b370-a31024eb9636\") " Jul 20 13:29:13.331394 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.331341 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj" (OuterVolumeSpecName: "kube-api-access-dzbqj") pod "19c90571-a3ac-4d34-b370-a31024eb9636" (UID: "19c90571-a3ac-4d34-b370-a31024eb9636"). InnerVolumeSpecName "kube-api-access-dzbqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:29:13.429974 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.429940 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzbqj\" (UniqueName: \"kubernetes.io/projected/19c90571-a3ac-4d34-b370-a31024eb9636-kube-api-access-dzbqj\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:29:13.942762 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.942730 2575 generic.go:356] "Generic (PLEG): container finished" podID="19c90571-a3ac-4d34-b370-a31024eb9636" containerID="4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325" exitCode=0 Jul 20 13:29:13.943182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.942772 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jscr5" Jul 20 13:29:13.943182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.942819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jscr5" event={"ID":"19c90571-a3ac-4d34-b370-a31024eb9636","Type":"ContainerDied","Data":"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325"} Jul 20 13:29:13.943182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.942865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jscr5" event={"ID":"19c90571-a3ac-4d34-b370-a31024eb9636","Type":"ContainerDied","Data":"34ded4aafe33f81c6019fc57b3bf4761677a8b57631d6d5c951f2106489c17c9"} Jul 20 13:29:13.943182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.942886 2575 scope.go:117] "RemoveContainer" containerID="4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325" Jul 20 13:29:13.951111 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.951093 2575 scope.go:117] "RemoveContainer" containerID="4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325" Jul 20 13:29:13.951420 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:29:13.951401 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325\": container with ID starting with 4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325 not found: ID does not exist" containerID="4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325" Jul 20 13:29:13.951480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.951430 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325"} err="failed to get container status \"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325\": rpc error: code = NotFound desc = could not find container \"4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325\": container with ID starting with 4361e5e5bb98ef159f8b7448bec76ff92412e4a68ca9a0252367a008d762e325 not found: ID does not exist" Jul 20 13:29:13.959326 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.959300 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:13.963003 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:13.962972 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jscr5"] Jul 20 13:29:15.881984 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:15.881952 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c90571-a3ac-4d34-b370-a31024eb9636" path="/var/lib/kubelet/pods/19c90571-a3ac-4d34-b370-a31024eb9636/volumes" Jul 20 13:29:16.583837 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.583798 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:29:16.584032 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.584004 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nrxjg" podUID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" containerName="authorino" containerID="cri-o://75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472" gracePeriod=30 Jul 20 13:29:16.832297 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.832273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:29:16.858014 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.857927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84js7\" (UniqueName: \"kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7\") pod \"da02579f-2d75-4aff-bc51-2cdcd9042ba8\" (UID: \"da02579f-2d75-4aff-bc51-2cdcd9042ba8\") " Jul 20 13:29:16.860127 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.860096 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7" (OuterVolumeSpecName: "kube-api-access-84js7") pod "da02579f-2d75-4aff-bc51-2cdcd9042ba8" (UID: "da02579f-2d75-4aff-bc51-2cdcd9042ba8"). InnerVolumeSpecName "kube-api-access-84js7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:29:16.958232 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958186 2575 generic.go:356] "Generic (PLEG): container finished" podID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" containerID="75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472" exitCode=0 Jul 20 13:29:16.958747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nrxjg" Jul 20 13:29:16.958747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nrxjg" event={"ID":"da02579f-2d75-4aff-bc51-2cdcd9042ba8","Type":"ContainerDied","Data":"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472"} Jul 20 13:29:16.958747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nrxjg" event={"ID":"da02579f-2d75-4aff-bc51-2cdcd9042ba8","Type":"ContainerDied","Data":"3724ec8bbdbc1eb3f57f3af7b35cf599308ddd1b251fb044c24a3cf867a9e03a"} Jul 20 13:29:16.958747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958339 2575 scope.go:117] "RemoveContainer" containerID="75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472" Jul 20 13:29:16.958747 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.958540 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84js7\" (UniqueName: \"kubernetes.io/projected/da02579f-2d75-4aff-bc51-2cdcd9042ba8-kube-api-access-84js7\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:29:16.969982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.969963 2575 scope.go:117] "RemoveContainer" containerID="75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472" Jul 20 13:29:16.970264 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:29:16.970246 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472\": container with ID starting with 75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472 not found: ID does not exist" containerID="75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472" Jul 20 13:29:16.970325 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.970272 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472"} err="failed to get container status \"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472\": rpc error: code = NotFound desc = could not find container \"75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472\": container with ID starting with 75bfc2b2b80942d8c73e052463830305feededc4b31d43ab9df534203a5a5472 not found: ID does not exist" Jul 20 13:29:16.982089 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.982063 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:29:16.986176 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:16.986155 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nrxjg"] Jul 20 13:29:17.881789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:29:17.881753 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" path="/var/lib/kubelet/pods/da02579f-2d75-4aff-bc51-2cdcd9042ba8/volumes" Jul 20 13:30:51.647532 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647495 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x"] Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647843 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c90571-a3ac-4d34-b370-a31024eb9636" containerName="authorino" Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647854 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c90571-a3ac-4d34-b370-a31024eb9636" containerName="authorino" Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647870 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" containerName="authorino" Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647876 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" containerName="authorino" Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647926 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c90571-a3ac-4d34-b370-a31024eb9636" containerName="authorino" Jul 20 13:30:51.648044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.647935 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="da02579f-2d75-4aff-bc51-2cdcd9042ba8" containerName="authorino" Jul 20 13:30:51.651020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.651005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.654846 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.654819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 20 13:30:51.654984 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.654847 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f779n\"" Jul 20 13:30:51.654984 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.654820 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 13:30:51.654984 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.654855 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 13:30:51.662275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.662250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x"] Jul 20 13:30:51.755979 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.755937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.756152 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.756006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dvl\" (UniqueName: \"kubernetes.io/projected/eeb09e15-a5f3-4627-a674-fb8406054db5-kube-api-access-c2dvl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.756152 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.756087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.756152 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.756121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb09e15-a5f3-4627-a674-fb8406054db5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.756257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.756159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.756257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.756195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856712 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dvl\" (UniqueName: \"kubernetes.io/projected/eeb09e15-a5f3-4627-a674-fb8406054db5-kube-api-access-c2dvl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb09e15-a5f3-4627-a674-fb8406054db5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.856922 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.856908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.857187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.857167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.857246 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.857196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.857314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.857292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.858936 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.858913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb09e15-a5f3-4627-a674-fb8406054db5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.859260 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.859244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb09e15-a5f3-4627-a674-fb8406054db5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.865579 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.865556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dvl\" (UniqueName: \"kubernetes.io/projected/eeb09e15-a5f3-4627-a674-fb8406054db5-kube-api-access-c2dvl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tt56x\" (UID: \"eeb09e15-a5f3-4627-a674-fb8406054db5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:51.961168 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:51.961090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:30:52.104306 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:52.104283 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x"] Jul 20 13:30:52.107046 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:30:52.107017 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb09e15_a5f3_4627_a674_fb8406054db5.slice/crio-f8c466b8855cee88aadb56c4653101b76ae49fdfa9c974b4f1f8e95e6700b85b WatchSource:0}: Error finding container f8c466b8855cee88aadb56c4653101b76ae49fdfa9c974b4f1f8e95e6700b85b: Status 404 returned error can't find the container with id f8c466b8855cee88aadb56c4653101b76ae49fdfa9c974b4f1f8e95e6700b85b Jul 20 13:30:52.395186 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:52.395143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" event={"ID":"eeb09e15-a5f3-4627-a674-fb8406054db5","Type":"ContainerStarted","Data":"f8c466b8855cee88aadb56c4653101b76ae49fdfa9c974b4f1f8e95e6700b85b"} Jul 20 13:30:53.401335 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:53.401227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" event={"ID":"eeb09e15-a5f3-4627-a674-fb8406054db5","Type":"ContainerStarted","Data":"939332f04baa79c5ad6d9755c3efd9a46346f13a9c3e7c16b4f1cba12c8a75cb"} Jul 20 13:30:53.422719 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:53.422656 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" podStartSLOduration=1.385818285 podStartE2EDuration="2.422637014s" totalImagesPullingTime="1.036818729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:53.420688682 +0000 UTC m=+852.086627711" watchObservedRunningTime="2026-07-20 13:30:53.422637014 +0000 UTC m=+852.088576047" Jul 20 13:30:56.264417 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.264342 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm"] Jul 20 13:30:56.268302 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.268285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.271241 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.271220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 20 13:30:56.277629 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.277607 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm"] Jul 20 13:30:56.399161 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.399161 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.399444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc46f50-70e6-4072-a392-724703838449-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.399444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dc8c\" (UniqueName: \"kubernetes.io/projected/5fc46f50-70e6-4072-a392-724703838449-kube-api-access-2dc8c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.399444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.399444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.399300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500052 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.499997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc46f50-70e6-4072-a392-724703838449-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dc8c\" (UniqueName: \"kubernetes.io/projected/5fc46f50-70e6-4072-a392-724703838449-kube-api-access-2dc8c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500417 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500703 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500814 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500706 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.500814 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.500676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.502474 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.502446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc46f50-70e6-4072-a392-724703838449-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.502575 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.502556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc46f50-70e6-4072-a392-724703838449-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.509054 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.509023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dc8c\" (UniqueName: \"kubernetes.io/projected/5fc46f50-70e6-4072-a392-724703838449-kube-api-access-2dc8c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ktrdm\" (UID: \"5fc46f50-70e6-4072-a392-724703838449\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.579193 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.579095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:30:56.714012 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:56.713986 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm"] Jul 20 13:30:56.715403 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:30:56.715377 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc46f50_70e6_4072_a392_724703838449.slice/crio-50c2cb317761feafaaa20c91904f2891d8e63966177a4ab0e6dbc4809ce1cd08 WatchSource:0}: Error finding container 50c2cb317761feafaaa20c91904f2891d8e63966177a4ab0e6dbc4809ce1cd08: Status 404 returned error can't find the container with id 50c2cb317761feafaaa20c91904f2891d8e63966177a4ab0e6dbc4809ce1cd08 Jul 20 13:30:57.423014 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:57.422975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" event={"ID":"5fc46f50-70e6-4072-a392-724703838449","Type":"ContainerStarted","Data":"8cb485f103a56524bac8b58b63b9c8312c2efb3d9a798e63b469b12af0bc3865"} Jul 20 13:30:57.423448 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:57.423038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" event={"ID":"5fc46f50-70e6-4072-a392-724703838449","Type":"ContainerStarted","Data":"50c2cb317761feafaaa20c91904f2891d8e63966177a4ab0e6dbc4809ce1cd08"} Jul 20 13:30:57.440666 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:30:57.440608 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" podStartSLOduration=1.250650436 podStartE2EDuration="1.440591098s" totalImagesPullingTime="189.940662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:57.439634404 +0000 UTC m=+856.105573447" watchObservedRunningTime="2026-07-20 13:30:57.440591098 +0000 UTC m=+856.106530131" Jul 20 13:31:01.961996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:01.961963 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:31:01.962436 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:01.962021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:31:01.974474 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:01.974449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:31:02.456809 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:02.456782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tt56x" Jul 20 13:31:06.579675 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:06.579629 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:31:06.580116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:06.579708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:31:06.592120 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:06.592094 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:31:07.477923 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:07.477894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ktrdm" Jul 20 13:31:21.845663 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.845631 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm"] Jul 20 13:31:21.848783 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.848764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.851449 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.851428 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 20 13:31:21.860059 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.860032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm"] Jul 20 13:31:21.912258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.912258 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a4475b-8f87-41f6-9059-14ef1818f363-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.912533 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.912533 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.912533 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:21.912533 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:21.912490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqngz\" (UniqueName: \"kubernetes.io/projected/74a4475b-8f87-41f6-9059-14ef1818f363-kube-api-access-zqngz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.012929 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.012894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013159 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.012941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a4475b-8f87-41f6-9059-14ef1818f363-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013159 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.012974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013159 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013159 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013159 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqngz\" (UniqueName: \"kubernetes.io/projected/74a4475b-8f87-41f6-9059-14ef1818f363-kube-api-access-zqngz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013868 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.013868 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.013834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.015986 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.015958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a4475b-8f87-41f6-9059-14ef1818f363-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.016187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.016169 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a4475b-8f87-41f6-9059-14ef1818f363-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.021908 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.021878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqngz\" (UniqueName: \"kubernetes.io/projected/74a4475b-8f87-41f6-9059-14ef1818f363-kube-api-access-zqngz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm\" (UID: \"74a4475b-8f87-41f6-9059-14ef1818f363\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.158465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.158424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:22.315334 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.315301 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm"] Jul 20 13:31:22.318987 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:31:22.318954 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74a4475b_8f87_41f6_9059_14ef1818f363.slice/crio-2476670ae93f6cf145326f4e3a7c59d3b203f3df9aa128c414ea450dbdcefa34 WatchSource:0}: Error finding container 2476670ae93f6cf145326f4e3a7c59d3b203f3df9aa128c414ea450dbdcefa34: Status 404 returned error can't find the container with id 2476670ae93f6cf145326f4e3a7c59d3b203f3df9aa128c414ea450dbdcefa34 Jul 20 13:31:22.538674 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:22.538588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" event={"ID":"74a4475b-8f87-41f6-9059-14ef1818f363","Type":"ContainerStarted","Data":"2476670ae93f6cf145326f4e3a7c59d3b203f3df9aa128c414ea450dbdcefa34"} Jul 20 13:31:24.551678 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:24.551636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" event={"ID":"74a4475b-8f87-41f6-9059-14ef1818f363","Type":"ContainerStarted","Data":"4d7eb0523bb2a25d0e1d963dbf835ba14be17c38bb0ce4e5792d34f530cd899f"} Jul 20 13:31:24.571765 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:24.571712 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" podStartSLOduration=1.9742025600000002 podStartE2EDuration="3.571694982s" totalImagesPullingTime="1.597492422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:31:24.569999674 +0000 UTC m=+883.235938706" watchObservedRunningTime="2026-07-20 13:31:24.571694982 +0000 UTC m=+883.237634012" Jul 20 13:31:32.159414 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:32.159376 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:32.159873 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:32.159429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:32.171907 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:32.171883 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:31:32.599286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:31:32.599198 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm" Jul 20 13:32:44.547452 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.547413 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd4b9b4f4-9xlcm"] Jul 20 13:32:44.550816 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.550799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.555261 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.555235 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-94vrh\"" Jul 20 13:32:44.555413 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.555284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 13:32:44.555413 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.555237 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 13:32:44.555413 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.555292 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 13:32:44.561043 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.561022 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd4b9b4f4-9xlcm"] Jul 20 13:32:44.664369 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.664328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-tls-cert\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.664545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.664447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5lw\" (UniqueName: \"kubernetes.io/projected/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-kube-api-access-hm5lw\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.765692 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.765653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5lw\" (UniqueName: \"kubernetes.io/projected/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-kube-api-access-hm5lw\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.765840 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.765792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-tls-cert\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.768346 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.768321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-tls-cert\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.774655 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.774632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5lw\" (UniqueName: \"kubernetes.io/projected/7a27a718-1c81-4cc0-80e0-b38ea84e9be4-kube-api-access-hm5lw\") pod \"authorino-bd4b9b4f4-9xlcm\" (UID: \"7a27a718-1c81-4cc0-80e0-b38ea84e9be4\") " pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:44.860192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:44.860099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" Jul 20 13:32:45.002236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:45.002213 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd4b9b4f4-9xlcm"] Jul 20 13:32:45.004451 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:32:45.004407 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a27a718_1c81_4cc0_80e0_b38ea84e9be4.slice/crio-c04614c6a3df2dce9fc21039ac7c13c2c6f95a209a62cd7a6cd9f3a8b33183d6 WatchSource:0}: Error finding container c04614c6a3df2dce9fc21039ac7c13c2c6f95a209a62cd7a6cd9f3a8b33183d6: Status 404 returned error can't find the container with id c04614c6a3df2dce9fc21039ac7c13c2c6f95a209a62cd7a6cd9f3a8b33183d6 Jul 20 13:32:45.919477 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:45.919436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" event={"ID":"7a27a718-1c81-4cc0-80e0-b38ea84e9be4","Type":"ContainerStarted","Data":"150bc47831446b316cd1a152752d9dfdeeabe2199212b5b728dcc82fb76927d0"} Jul 20 13:32:45.919993 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:45.919486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" event={"ID":"7a27a718-1c81-4cc0-80e0-b38ea84e9be4","Type":"ContainerStarted","Data":"c04614c6a3df2dce9fc21039ac7c13c2c6f95a209a62cd7a6cd9f3a8b33183d6"} Jul 20 13:32:45.940583 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:32:45.940471 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bd4b9b4f4-9xlcm" podStartSLOduration=1.274658909 podStartE2EDuration="1.940451832s" totalImagesPullingTime="665.792923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:32:45.938231681 +0000 UTC m=+964.604170713" watchObservedRunningTime="2026-07-20 13:32:45.940451832 +0000 UTC m=+964.606390862" Jul 20 13:43:42.466179 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.466048 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:43:42.469480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.469451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.472277 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.472248 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a934d7de-kserve-self-signed-certs\"" Jul 20 13:43:42.490851 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.490821 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:43:42.637615 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.637615 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.637856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.637856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.637856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.637856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.637834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2rt\" (UniqueName: \"kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.738930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2rt\" (UniqueName: \"kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739567 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739678 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.739678 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.739610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.741400 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.741376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.741735 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.741714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.751049 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.751025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2rt\" (UniqueName: \"kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt\") pod \"e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.779020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.778982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:42.919102 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:42.919074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:43:42.920722 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:43:42.920695 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13bf2229_5815_4eea_95d1_edaed5e7e8b6.slice/crio-5bd87b1bc4e3a35f4052a919ecc57427bbcaf795115a23eefaf303e828c10dbd WatchSource:0}: Error finding container 5bd87b1bc4e3a35f4052a919ecc57427bbcaf795115a23eefaf303e828c10dbd: Status 404 returned error can't find the container with id 5bd87b1bc4e3a35f4052a919ecc57427bbcaf795115a23eefaf303e828c10dbd Jul 20 13:43:43.031906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:43.031810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" event={"ID":"13bf2229-5815-4eea-95d1-edaed5e7e8b6","Type":"ContainerStarted","Data":"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68"} Jul 20 13:43:43.031906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:43.031854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" event={"ID":"13bf2229-5815-4eea-95d1-edaed5e7e8b6","Type":"ContainerStarted","Data":"5bd87b1bc4e3a35f4052a919ecc57427bbcaf795115a23eefaf303e828c10dbd"} Jul 20 13:43:43.051438 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:43.051380 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" podStartSLOduration=1.051347134 podStartE2EDuration="1.051347134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:43:43.05107774 +0000 UTC m=+1621.717016771" watchObservedRunningTime="2026-07-20 13:43:43.051347134 +0000 UTC m=+1621.717286165" Jul 20 13:43:49.548832 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.548791 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:43:49.552400 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.552382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.556410 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.556389 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a934d7de-kserve-self-signed-certs\"" Jul 20 13:43:49.571260 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.571237 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:43:49.603441 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.603620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.603620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.603620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.603620 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9mj\" (UniqueName: \"kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.603843 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.603628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.704721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.704686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.704886 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.704735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.704886 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.704766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.704886 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.704796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9mj\" (UniqueName: \"kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.705008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.704936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.705077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.705060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.705129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.705113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.705254 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.705232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.705490 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.705467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.707047 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.707025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.707302 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.707284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.714781 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.714759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9mj\" (UniqueName: \"kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj\") pod \"e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:49.861792 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:49.861700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:50.008085 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:50.008059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:43:50.010173 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:43:50.010143 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5905b506_9064_4f4a_80c0_b80b60cc74e2.slice/crio-017a785623e40dd539f3474662154905b6849f94b0166e7ebc78e61091423b9b WatchSource:0}: Error finding container 017a785623e40dd539f3474662154905b6849f94b0166e7ebc78e61091423b9b: Status 404 returned error can't find the container with id 017a785623e40dd539f3474662154905b6849f94b0166e7ebc78e61091423b9b Jul 20 13:43:50.071008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:50.070970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" event={"ID":"5905b506-9064-4f4a-80c0-b80b60cc74e2","Type":"ContainerStarted","Data":"017a785623e40dd539f3474662154905b6849f94b0166e7ebc78e61091423b9b"} Jul 20 13:43:51.077421 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:51.077379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" event={"ID":"5905b506-9064-4f4a-80c0-b80b60cc74e2","Type":"ContainerStarted","Data":"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863"} Jul 20 13:43:51.098231 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:51.098179 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" podStartSLOduration=2.098162491 podStartE2EDuration="2.098162491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:43:51.097134061 +0000 UTC m=+1629.763073093" watchObservedRunningTime="2026-07-20 13:43:51.098162491 +0000 UTC m=+1629.764101521" Jul 20 13:43:52.780071 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:52.779979 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:52.780071 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:52.780037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:52.792493 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:52.792471 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:53.097627 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:53.097534 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:43:59.862268 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:59.862225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:59.862753 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:59.862282 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:43:59.874624 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:43:59.874600 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:44:00.133281 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:00.133256 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:44:24.905685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:24.905653 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:44:25.786545 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:25.786513 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:44:39.925618 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:39.925580 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" podUID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" containerName="main" containerID="cri-o://bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68" gracePeriod=45 Jul 20 13:44:40.184118 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.184056 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:44:40.291977 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.291943 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.291997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292117 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz2rt\" (UniqueName: \"kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292157 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292135 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir\") pod \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\" (UID: \"13bf2229-5815-4eea-95d1-edaed5e7e8b6\") " Jul 20 13:44:40.292450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292278 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:40.292450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292324 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home" (OuterVolumeSpecName: "home") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:40.292450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292392 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.292587 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.292484 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:40.294287 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.294257 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm" (OuterVolumeSpecName: "dshm") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:40.294420 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.294300 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:44:40.294420 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.294322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt" (OuterVolumeSpecName: "kube-api-access-wz2rt") pod "13bf2229-5815-4eea-95d1-edaed5e7e8b6" (UID: "13bf2229-5815-4eea-95d1-edaed5e7e8b6"). InnerVolumeSpecName "kube-api-access-wz2rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:44:40.326629 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.326591 2575 generic.go:356] "Generic (PLEG): container finished" podID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" containerID="bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68" exitCode=0 Jul 20 13:44:40.326787 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.326651 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" Jul 20 13:44:40.326787 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.326653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" event={"ID":"13bf2229-5815-4eea-95d1-edaed5e7e8b6","Type":"ContainerDied","Data":"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68"} Jul 20 13:44:40.326787 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.326759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26" event={"ID":"13bf2229-5815-4eea-95d1-edaed5e7e8b6","Type":"ContainerDied","Data":"5bd87b1bc4e3a35f4052a919ecc57427bbcaf795115a23eefaf303e828c10dbd"} Jul 20 13:44:40.326787 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.326778 2575 scope.go:117] "RemoveContainer" containerID="bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68" Jul 20 13:44:40.335820 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.335801 2575 scope.go:117] "RemoveContainer" containerID="bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68" Jul 20 13:44:40.336067 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:44:40.336048 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68\": container with ID starting with bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68 not found: ID does not exist" containerID="bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68" Jul 20 13:44:40.336129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.336074 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68"} err="failed to get container status \"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68\": rpc error: code = NotFound desc = could not find container \"bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68\": container with ID starting with bff3fd6cc4e0bd98fe7a1927be3f30e227e22baae5ac6543d041ee938e524d68 not found: ID does not exist" Jul 20 13:44:40.363583 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.363553 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:44:40.365827 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.365800 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a934d7de-kserve-d977cbf4c-tsv26"] Jul 20 13:44:40.393082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.393058 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.393082 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.393083 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz2rt\" (UniqueName: \"kubernetes.io/projected/13bf2229-5815-4eea-95d1-edaed5e7e8b6-kube-api-access-wz2rt\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.393235 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.393093 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.393235 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.393103 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.393235 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.393111 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bf2229-5815-4eea-95d1-edaed5e7e8b6-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:40.806442 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:40.806407 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" podUID="5905b506-9064-4f4a-80c0-b80b60cc74e2" containerName="main" containerID="cri-o://1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863" gracePeriod=45 Jul 20 13:44:41.061195 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.061136 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:44:41.201186 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201147 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201415 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201195 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201415 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201260 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201415 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201286 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201415 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201415 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb9mj\" (UniqueName: \"kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj\") pod \"5905b506-9064-4f4a-80c0-b80b60cc74e2\" (UID: \"5905b506-9064-4f4a-80c0-b80b60cc74e2\") " Jul 20 13:44:41.201676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201444 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:41.201676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201542 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home" (OuterVolumeSpecName: "home") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:41.201676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201572 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:41.202056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201733 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.202056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201779 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.202056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.201793 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.203543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.203518 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:44:41.203543 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.203532 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm" (OuterVolumeSpecName: "dshm") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:44:41.203684 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.203576 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj" (OuterVolumeSpecName: "kube-api-access-jb9mj") pod "5905b506-9064-4f4a-80c0-b80b60cc74e2" (UID: "5905b506-9064-4f4a-80c0-b80b60cc74e2"). InnerVolumeSpecName "kube-api-access-jb9mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:44:41.302234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.302194 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5905b506-9064-4f4a-80c0-b80b60cc74e2-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.302234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.302221 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5905b506-9064-4f4a-80c0-b80b60cc74e2-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.302234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.302239 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9mj\" (UniqueName: \"kubernetes.io/projected/5905b506-9064-4f4a-80c0-b80b60cc74e2-kube-api-access-jb9mj\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:44:41.334421 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.334318 2575 generic.go:356] "Generic (PLEG): container finished" podID="5905b506-9064-4f4a-80c0-b80b60cc74e2" containerID="1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863" exitCode=0 Jul 20 13:44:41.334421 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.334392 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" Jul 20 13:44:41.334618 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.334388 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" event={"ID":"5905b506-9064-4f4a-80c0-b80b60cc74e2","Type":"ContainerDied","Data":"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863"} Jul 20 13:44:41.334618 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.334502 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4" event={"ID":"5905b506-9064-4f4a-80c0-b80b60cc74e2","Type":"ContainerDied","Data":"017a785623e40dd539f3474662154905b6849f94b0166e7ebc78e61091423b9b"} Jul 20 13:44:41.334618 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.334522 2575 scope.go:117] "RemoveContainer" containerID="1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863" Jul 20 13:44:41.343580 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.343562 2575 scope.go:117] "RemoveContainer" containerID="1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863" Jul 20 13:44:41.343820 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:44:41.343801 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863\": container with ID starting with 1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863 not found: ID does not exist" containerID="1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863" Jul 20 13:44:41.343875 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.343828 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863"} err="failed to get container status \"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863\": rpc error: code = NotFound desc = could not find container \"1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863\": container with ID starting with 1a9e43b59d976f8801392a6ec0861898f34addd339a67a184c4a9052acf4b863 not found: ID does not exist" Jul 20 13:44:41.356658 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.356635 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:44:41.362879 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.362859 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a934d7de-kserve-6c7fccf48c-pwcl4"] Jul 20 13:44:41.881286 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.881258 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" path="/var/lib/kubelet/pods/13bf2229-5815-4eea-95d1-edaed5e7e8b6/volumes" Jul 20 13:44:41.881699 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:44:41.881686 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5905b506-9064-4f4a-80c0-b80b60cc74e2" path="/var/lib/kubelet/pods/5905b506-9064-4f4a-80c0-b80b60cc74e2/volumes" Jul 20 13:49:04.919305 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919273 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:04.919789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919719 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" containerName="main" Jul 20 13:49:04.919789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919731 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" containerName="main" Jul 20 13:49:04.919789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919738 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5905b506-9064-4f4a-80c0-b80b60cc74e2" containerName="main" Jul 20 13:49:04.919789 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919744 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5905b506-9064-4f4a-80c0-b80b60cc74e2" containerName="main" Jul 20 13:49:04.919923 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919802 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5905b506-9064-4f4a-80c0-b80b60cc74e2" containerName="main" Jul 20 13:49:04.919923 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.919811 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13bf2229-5815-4eea-95d1-edaed5e7e8b6" containerName="main" Jul 20 13:49:04.922702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.922686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:04.928348 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.926470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d5814763-openshift-default-dockercfg-pgkx7\"" Jul 20 13:49:04.946605 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:04.946574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:05.025366 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025558 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025558 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025558 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhk2\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.025738 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.025709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126480 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhk2\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.126961 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.126783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127079 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127466 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127466 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.127466 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.127432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.128965 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.128941 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.129066 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.129050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.139254 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.139224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.139416 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.139260 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhk2\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2\") pod \"e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.240439 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.240323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:05.386784 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.386756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:05.387849 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:49:05.387826 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28f10ca_87ea_414f_ba0c_9de428bc11fe.slice/crio-af5134295f8f073585fe5ba5014ab597be82ce2efbb07c34c5f5032e22781840 WatchSource:0}: Error finding container af5134295f8f073585fe5ba5014ab597be82ce2efbb07c34c5f5032e22781840: Status 404 returned error can't find the container with id af5134295f8f073585fe5ba5014ab597be82ce2efbb07c34c5f5032e22781840 Jul 20 13:49:05.389991 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.389954 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:05.390054 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.390039 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:05.390093 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.390079 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:05.599249 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.599157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerStarted","Data":"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4"} Jul 20 13:49:05.599249 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.599220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerStarted","Data":"af5134295f8f073585fe5ba5014ab597be82ce2efbb07c34c5f5032e22781840"} Jul 20 13:49:05.623408 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:05.623331 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podStartSLOduration=1.623314953 podStartE2EDuration="1.623314953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:49:05.621526237 +0000 UTC m=+1944.287465263" watchObservedRunningTime="2026-07-20 13:49:05.623314953 +0000 UTC m=+1944.289253984" Jul 20 13:49:06.241516 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:06.241476 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:06.246430 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:06.246403 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:06.604507 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:06.604415 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:06.605275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:06.605256 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:19.668882 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.668846 2575 generic.go:356] "Generic (PLEG): container finished" podID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerID="749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4" exitCode=0 Jul 20 13:49:19.669319 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.668892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerDied","Data":"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4"} Jul 20 13:49:19.669319 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.669279 2575 scope.go:117] "RemoveContainer" containerID="749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4" Jul 20 13:49:19.670256 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.670227 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:19.670315 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.670301 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:19.670374 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:19.670338 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:49:20.675835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:20.675798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerStarted","Data":"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638"} Jul 20 13:49:21.241592 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:21.241552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:21.243212 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:21.243174 2575 patch_prober.go:28] interesting pod/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 13:49:21.243407 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:21.243242 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 13:49:21.604569 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:21.604468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:22.241330 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:22.241292 2575 patch_prober.go:28] interesting pod/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 13:49:22.241844 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:22.241388 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 13:49:23.241649 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:23.241611 2575 patch_prober.go:28] interesting pod/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 13:49:23.242250 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:23.242203 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 13:49:24.241694 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:24.241653 2575 patch_prober.go:28] interesting pod/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 13:49:24.242220 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:24.241733 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 13:49:25.244906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:25.244872 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:25.245916 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:25.245898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:47.755418 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:47.755294 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:47.755999 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:47.755597 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" containerID="cri-o://a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638" gracePeriod=30 Jul 20 13:49:53.017720 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.017655 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:53.170424 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170390 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170433 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqhk2\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170520 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170555 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170610 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170594 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170903 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.170903 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170704 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo\") pod \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\" (UID: \"e28f10ca-87ea-414f-ba0c-9de428bc11fe\") " Jul 20 13:49:53.171015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.170920 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:49:53.171100 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.171069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:49:53.171167 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.171095 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.171313 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.171286 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:49:53.171313 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.171296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:49:53.171529 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.171386 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:49:53.172930 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.172907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:49:53.173026 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.172923 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2" (OuterVolumeSpecName: "kube-api-access-rqhk2") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "kube-api-access-rqhk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:49:53.173121 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.173094 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:49:53.173236 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.173145 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e28f10ca-87ea-414f-ba0c-9de428bc11fe" (UID: "e28f10ca-87ea-414f-ba0c-9de428bc11fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272059 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272090 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272099 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272107 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqhk2\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-kube-api-access-rqhk2\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272118 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272126 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f10ca-87ea-414f-ba0c-9de428bc11fe-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272133 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.272150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.272141 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f10ca-87ea-414f-ba0c-9de428bc11fe-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:49:53.832112 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.832076 2575 generic.go:356] "Generic (PLEG): container finished" podID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerID="a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638" exitCode=0 Jul 20 13:49:53.832284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.832122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerDied","Data":"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638"} Jul 20 13:49:53.832284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.832130 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" Jul 20 13:49:53.832284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.832173 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj" event={"ID":"e28f10ca-87ea-414f-ba0c-9de428bc11fe","Type":"ContainerDied","Data":"af5134295f8f073585fe5ba5014ab597be82ce2efbb07c34c5f5032e22781840"} Jul 20 13:49:53.832284 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.832198 2575 scope.go:117] "RemoveContainer" containerID="a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638" Jul 20 13:49:53.841572 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.841552 2575 scope.go:117] "RemoveContainer" containerID="749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4" Jul 20 13:49:53.849508 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.849490 2575 scope.go:117] "RemoveContainer" containerID="a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638" Jul 20 13:49:53.849775 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:49:53.849757 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638\": container with ID starting with a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638 not found: ID does not exist" containerID="a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638" Jul 20 13:49:53.849822 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.849785 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638"} err="failed to get container status \"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638\": rpc error: code = NotFound desc = could not find container \"a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638\": container with ID starting with a7efbe47969131a95802c304f642fed36cca209304fb3597f89f90a6e244f638 not found: ID does not exist" Jul 20 13:49:53.849822 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.849804 2575 scope.go:117] "RemoveContainer" containerID="749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4" Jul 20 13:49:53.850025 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:49:53.850006 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4\": container with ID starting with 749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4 not found: ID does not exist" containerID="749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4" Jul 20 13:49:53.850088 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.850036 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4"} err="failed to get container status \"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4\": rpc error: code = NotFound desc = could not find container \"749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4\": container with ID starting with 749559fae7af9967c0d599dd84abeb4805d72a7223bb4835c0516bf75f5460d4 not found: ID does not exist" Jul 20 13:49:53.854549 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.854523 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:53.858500 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.858480 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d5814763-openshift-default-6ccf648fff-mrbzj"] Jul 20 13:49:53.881505 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:49:53.881481 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" path="/var/lib/kubelet/pods/e28f10ca-87ea-414f-ba0c-9de428bc11fe/volumes" Jul 20 13:50:32.265724 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.265690 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:50:32.266328 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.266060 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:50:32.266328 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.266073 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:50:32.266328 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.266153 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:50:32.266328 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.266159 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:50:32.269405 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.269384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.272916 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.272890 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-358b8783-openshift-default-dockercfg-msf6b\"" Jul 20 13:50:32.285484 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.285456 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:50:32.339934 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.339903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340135 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.339956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340135 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340135 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340135 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340376 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340376 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5rf\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340376 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.340376 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.340302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441659 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.441867 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5rf\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.441916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.442111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.442143 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442192 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.442150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442444 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.442422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.442590 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.442568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.444316 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.444288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.444596 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.444579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.450669 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.450639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.450852 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.450836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5rf\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf\") pod \"e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.581163 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.581068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:32.722275 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.722002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:50:32.728682 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.728638 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:32.728770 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.728726 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:32.728770 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:32.728757 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:33.021269 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:33.021221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerStarted","Data":"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3"} Jul 20 13:50:33.021493 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:33.021284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerStarted","Data":"6fafe78f6c287fac8e88faf15647f7b15f07692196b9768b04fa9fa59aaa625b"} Jul 20 13:50:33.043696 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:33.043648 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podStartSLOduration=1.043632256 podStartE2EDuration="1.043632256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:50:33.042581441 +0000 UTC m=+2031.708520473" watchObservedRunningTime="2026-07-20 13:50:33.043632256 +0000 UTC m=+2031.709571287" Jul 20 13:50:33.581430 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:33.581390 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:33.586322 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:33.586297 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:34.026496 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:34.026466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:34.027482 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:34.027465 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:49.276324 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:49.276277 2575 patch_prober.go:28] interesting pod/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:46680->10.133.0.42:15021: read: connection reset by peer" start-of-body= Jul 20 13:50:49.276745 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:49.276596 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:46680->10.133.0.42:15021: read: connection reset by peer" Jul 20 13:50:50.104584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.104550 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerID="976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3" exitCode=0 Jul 20 13:50:50.104783 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.104602 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerDied","Data":"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3"} Jul 20 13:50:50.105036 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.105018 2575 scope.go:117] "RemoveContainer" containerID="976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3" Jul 20 13:50:50.106141 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.106109 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:50.106227 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.106196 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:50.106278 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:50.106260 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:50:51.111702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:51.111667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerStarted","Data":"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0"} Jul 20 13:50:51.581749 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:51.581704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:51.581982 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:51.581957 2575 patch_prober.go:28] interesting pod/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 20 13:50:51.582030 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:51.582012 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 20 13:50:52.581732 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:52.581698 2575 patch_prober.go:28] interesting pod/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 20 13:50:52.582138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:52.581752 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 20 13:50:53.582371 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:53.582333 2575 patch_prober.go:28] interesting pod/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 20 13:50:53.582772 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:53.582414 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 20 13:50:54.581831 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:54.581787 2575 patch_prober.go:28] interesting pod/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 20 13:50:54.582055 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:54.581852 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 20 13:50:55.585228 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:55.585195 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:55.585727 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:55.585617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:50:55.586110 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:50:55.586090 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:51:16.694044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:16.694012 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:51:16.694597 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:16.694284 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" containerID="cri-o://bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0" gracePeriod=30 Jul 20 13:51:21.952517 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:21.952487 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:51:22.121716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121607 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.121716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121674 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.121716 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp5rf\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121795 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121817 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121843 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122007 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.121926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy\") pod \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\" (UID: \"bf6f784f-ddc8-48bb-860d-8948bdf29bb8\") " Jul 20 13:51:22.122314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122094 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:51:22.122314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122111 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:51:22.122314 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:51:22.122588 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122450 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.122588 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122474 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.122588 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122490 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.122872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.122849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:51:22.123044 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.123016 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:51:22.124156 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.124132 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:51:22.124259 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.124178 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf" (OuterVolumeSpecName: "kube-api-access-gp5rf") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "kube-api-access-gp5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:51:22.124530 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.124507 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:51:22.124530 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.124522 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf6f784f-ddc8-48bb-860d-8948bdf29bb8" (UID: "bf6f784f-ddc8-48bb-860d-8948bdf29bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:51:22.223399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223341 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.223399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223393 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.223399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223402 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.223399 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223409 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.223676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223418 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.223676 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.223427 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp5rf\" (UniqueName: \"kubernetes.io/projected/bf6f784f-ddc8-48bb-860d-8948bdf29bb8-kube-api-access-gp5rf\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:51:22.261056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.261021 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerID="bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0" exitCode=0 Jul 20 13:51:22.261223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.261075 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" Jul 20 13:51:22.261223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.261109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerDied","Data":"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0"} Jul 20 13:51:22.261223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.261166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw" event={"ID":"bf6f784f-ddc8-48bb-860d-8948bdf29bb8","Type":"ContainerDied","Data":"6fafe78f6c287fac8e88faf15647f7b15f07692196b9768b04fa9fa59aaa625b"} Jul 20 13:51:22.261223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.261188 2575 scope.go:117] "RemoveContainer" containerID="bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0" Jul 20 13:51:22.278263 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.278237 2575 scope.go:117] "RemoveContainer" containerID="976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3" Jul 20 13:51:22.286539 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.286521 2575 scope.go:117] "RemoveContainer" containerID="bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0" Jul 20 13:51:22.286793 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:51:22.286772 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0\": container with ID starting with bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0 not found: ID does not exist" containerID="bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0" Jul 20 13:51:22.286842 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.286801 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0"} err="failed to get container status \"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0\": rpc error: code = NotFound desc = could not find container \"bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0\": container with ID starting with bc680742daa485c43dff89207cb95216f68ec3b1e59cede171e3941c6e0be7d0 not found: ID does not exist" Jul 20 13:51:22.286842 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.286818 2575 scope.go:117] "RemoveContainer" containerID="976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3" Jul 20 13:51:22.287041 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:51:22.287023 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3\": container with ID starting with 976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3 not found: ID does not exist" containerID="976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3" Jul 20 13:51:22.287078 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.287048 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3"} err="failed to get container status \"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3\": rpc error: code = NotFound desc = could not find container \"976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3\": container with ID starting with 976e6243d8ea918ea68fcd7bb7ea7c6838ba2d4a2e67c19caa9ee9c6b2a62eb3 not found: ID does not exist" Jul 20 13:51:22.293584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.293563 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:51:22.299110 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:22.299087 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-358b8783-openshift-default-6cb9f64df9-9wzsw"] Jul 20 13:51:23.882893 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:51:23.882860 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" path="/var/lib/kubelet/pods/bf6f784f-ddc8-48bb-860d-8948bdf29bb8/volumes" Jul 20 13:54:34.708013 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.707932 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:54:34.708522 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708505 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.708599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708525 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.708599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708538 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.708599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708547 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.708599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708559 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:54:34.708599 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708568 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28f10ca-87ea-414f-ba0c-9de428bc11fe" containerName="istio-proxy" Jul 20 13:54:34.708759 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708666 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.708759 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.708678 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6f784f-ddc8-48bb-860d-8948bdf29bb8" containerName="istio-proxy" Jul 20 13:54:34.712207 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.712187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.715020 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.714986 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a45cad8-openshift-default-dockercfg-jqtdk\"" Jul 20 13:54:34.724538 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.724509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:54:34.810184 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810386 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpf5r\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810386 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810386 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810598 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.810598 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.810584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.911859 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.911811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.911859 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.911849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912126 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.911872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912126 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.911997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpf5r\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912126 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912126 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912614 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912614 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.912945 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.912924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.914371 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.914331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.914566 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.914550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.925059 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.925029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpf5r\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:34.925311 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:34.925294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token\") pod \"e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:35.023516 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.023429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:35.159104 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.158835 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:54:35.161079 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:54:35.161046 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafda0b32_96d2_430b_ae84_afb262a74141.slice/crio-31851c88092acba73c6bb7f0cf267155bee87bb3cbf9e50f4461d883523a5a14 WatchSource:0}: Error finding container 31851c88092acba73c6bb7f0cf267155bee87bb3cbf9e50f4461d883523a5a14: Status 404 returned error can't find the container with id 31851c88092acba73c6bb7f0cf267155bee87bb3cbf9e50f4461d883523a5a14 Jul 20 13:54:35.163090 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.163057 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:35.163169 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.163122 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:35.163169 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.163152 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:35.179062 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:35.179037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerStarted","Data":"31851c88092acba73c6bb7f0cf267155bee87bb3cbf9e50f4461d883523a5a14"} Jul 20 13:54:36.184853 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:36.184817 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerStarted","Data":"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9"} Jul 20 13:54:36.210342 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:36.210283 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podStartSLOduration=2.210266145 podStartE2EDuration="2.210266145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:54:36.208501845 +0000 UTC m=+2274.874440878" watchObservedRunningTime="2026-07-20 13:54:36.210266145 +0000 UTC m=+2274.876205176" Jul 20 13:54:37.023892 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:37.023851 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:37.028669 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:37.028639 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:37.189639 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:37.189610 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:37.190722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:37.190701 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:48.649829 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.649785 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:54:48.663615 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.663575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:54:48.663803 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.663696 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.671396 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.670395 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a45cad8\"/\"default-dockercfg-68knz\"" Jul 20 13:54:48.675722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.675680 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a45cad8\"/\"openshift-service-ca.crt\"" Jul 20 13:54:48.675722 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.675686 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a45cad8\"/\"kube-root-ca.crt\"" Jul 20 13:54:48.675919 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.675774 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a45cad8\"/\"e2e-lifecycle-model-6a45cad8-kserve-self-signed-certs\"" Jul 20 13:54:48.741008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.740976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.741008 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.741018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.741243 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.741097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46782\" (UniqueName: \"kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.741243 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.741139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.741243 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.741182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.741243 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.741210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.842038 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.841989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.842338 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.842315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.842602 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.842583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46782\" (UniqueName: \"kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843129 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.843108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843374 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.843338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843565 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.843549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843713 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.843626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843814 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.842922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.843963 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.843923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.846317 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.846289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.850918 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.850879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.853501 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.853464 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46782\" (UniqueName: \"kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782\") pod \"e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:48.982868 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:48.982772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:49.141132 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:49.141109 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:54:49.266443 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:49.266108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" event={"ID":"b4999954-5d08-4bf1-badd-ee8b2e533a34","Type":"ContainerStarted","Data":"c9b2195192b7fadbfb44cb0a1b647732b0a57e94008dc9c2323a663f8ef2c5cb"} Jul 20 13:54:49.266443 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:49.266161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" event={"ID":"b4999954-5d08-4bf1-badd-ee8b2e533a34","Type":"ContainerStarted","Data":"264a282efc583d6364f8b877cc698e7a939d97e7ea57123b120d71a24aa37f1b"} Jul 20 13:54:49.287416 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:49.287332 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" podStartSLOduration=1.287315506 podStartE2EDuration="1.287315506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:54:49.284884126 +0000 UTC m=+2287.950823158" watchObservedRunningTime="2026-07-20 13:54:49.287315506 +0000 UTC m=+2287.953254528" Jul 20 13:54:52.687270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:52.686538 2575 patch_prober.go:28] interesting pod/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:60322->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 20 13:54:52.687270 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:52.686606 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:60322->10.133.0.43:15021: read: connection reset by peer" Jul 20 13:54:53.287917 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.287885 2575 generic.go:356] "Generic (PLEG): container finished" podID="afda0b32-96d2-430b-ae84-afb262a74141" containerID="1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9" exitCode=0 Jul 20 13:54:53.288093 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.287929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerDied","Data":"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9"} Jul 20 13:54:53.288387 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.288371 2575 scope.go:117] "RemoveContainer" containerID="1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9" Jul 20 13:54:53.289435 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.289403 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:53.289507 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.289494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:53.289544 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:53.289531 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:54:54.294911 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:54.294878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerStarted","Data":"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165"} Jul 20 13:54:55.024261 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:55.024222 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:55.024470 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:55.024434 2575 patch_prober.go:28] interesting pod/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 13:54:55.024519 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:55.024482 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 13:54:56.024087 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:56.024042 2575 patch_prober.go:28] interesting pod/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 13:54:56.024625 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:56.024116 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 13:54:57.024406 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:57.024341 2575 patch_prober.go:28] interesting pod/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 13:54:57.024805 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:57.024454 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 13:54:58.024572 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:58.024519 2575 patch_prober.go:28] interesting pod/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 13:54:58.024996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:58.024603 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 13:54:58.983465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:58.983429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:58.983680 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:58.983622 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:58.996261 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:58.996239 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:54:59.027206 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:59.027180 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:59.027618 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:59.027468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:59.028223 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:59.028204 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:54:59.332785 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:54:59.332703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:55:38.343908 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:38.343821 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:55:53.364136 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.364086 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" podUID="b4999954-5d08-4bf1-badd-ee8b2e533a34" containerName="main" containerID="cri-o://c9b2195192b7fadbfb44cb0a1b647732b0a57e94008dc9c2323a663f8ef2c5cb" gracePeriod=45 Jul 20 13:55:53.594395 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.594344 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4999954-5d08-4bf1-badd-ee8b2e533a34" containerID="c9b2195192b7fadbfb44cb0a1b647732b0a57e94008dc9c2323a663f8ef2c5cb" exitCode=0 Jul 20 13:55:53.594395 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.594387 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" event={"ID":"b4999954-5d08-4bf1-badd-ee8b2e533a34","Type":"ContainerDied","Data":"c9b2195192b7fadbfb44cb0a1b647732b0a57e94008dc9c2323a663f8ef2c5cb"} Jul 20 13:55:53.614547 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.614479 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:55:53.793903 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.793871 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.793918 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.793947 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.793990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46782\" (UniqueName: \"kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794077 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794312 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794100 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache\") pod \"b4999954-5d08-4bf1-badd-ee8b2e533a34\" (UID: \"b4999954-5d08-4bf1-badd-ee8b2e533a34\") " Jul 20 13:55:53.794312 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794197 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:55:53.794460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794346 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home" (OuterVolumeSpecName: "home") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:55:53.794460 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794441 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:55:53.794574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794556 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:53.794611 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794581 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:53.794658 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.794621 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:53.796249 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.796228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm" (OuterVolumeSpecName: "dshm") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:55:53.796309 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.796232 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:55:53.796346 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.796325 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782" (OuterVolumeSpecName: "kube-api-access-46782") pod "b4999954-5d08-4bf1-badd-ee8b2e533a34" (UID: "b4999954-5d08-4bf1-badd-ee8b2e533a34"). InnerVolumeSpecName "kube-api-access-46782". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:55:53.895524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.895497 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4999954-5d08-4bf1-badd-ee8b2e533a34-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:53.895524 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.895522 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46782\" (UniqueName: \"kubernetes.io/projected/b4999954-5d08-4bf1-badd-ee8b2e533a34-kube-api-access-46782\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:53.895695 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:53.895536 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4999954-5d08-4bf1-badd-ee8b2e533a34-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:55:54.600872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:54.600829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" event={"ID":"b4999954-5d08-4bf1-badd-ee8b2e533a34","Type":"ContainerDied","Data":"264a282efc583d6364f8b877cc698e7a939d97e7ea57123b120d71a24aa37f1b"} Jul 20 13:55:54.600872 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:54.600857 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd" Jul 20 13:55:54.601418 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:54.600896 2575 scope.go:117] "RemoveContainer" containerID="c9b2195192b7fadbfb44cb0a1b647732b0a57e94008dc9c2323a663f8ef2c5cb" Jul 20 13:55:54.617782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:54.617753 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:55:54.623771 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:54.623743 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6a45cad8/e2e-lifecycle-model-6a45cad8-kserve-566db55f59-9kqtd"] Jul 20 13:55:55.882906 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:55:55.882873 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4999954-5d08-4bf1-badd-ee8b2e533a34" path="/var/lib/kubelet/pods/b4999954-5d08-4bf1-badd-ee8b2e533a34/volumes" Jul 20 13:56:34.919820 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:34.919779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:56:34.920378 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:34.920029 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" containerID="cri-o://83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165" gracePeriod=30 Jul 20 13:56:40.180849 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.180820 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:56:40.243182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243182 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243207 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243334 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpf5r\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243417 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243465 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243430 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:56:40.243782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket\") pod \"afda0b32-96d2-430b-ae84-afb262a74141\" (UID: \"afda0b32-96d2-430b-ae84-afb262a74141\") " Jul 20 13:56:40.243782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:56:40.243782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243763 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.243782 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243779 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.243976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:56:40.243976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.243965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:56:40.244199 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.244122 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data" (OuterVolumeSpecName: "istio-data") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:56:40.245689 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.245658 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r" (OuterVolumeSpecName: "kube-api-access-fpf5r") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "kube-api-access-fpf5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:56:40.246023 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.245988 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token" (OuterVolumeSpecName: "istio-token") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:56:40.246118 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.246049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 13:56:40.246118 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.246062 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afda0b32-96d2-430b-ae84-afb262a74141" (UID: "afda0b32-96d2-430b-ae84-afb262a74141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:56:40.344484 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344444 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpf5r\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-kube-api-access-fpf5r\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344484 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344482 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afda0b32-96d2-430b-ae84-afb262a74141-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344500 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344514 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344526 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afda0b32-96d2-430b-ae84-afb262a74141-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344539 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afda0b32-96d2-430b-ae84-afb262a74141-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.344685 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.344550 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afda0b32-96d2-430b-ae84-afb262a74141-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:56:40.832675 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.832640 2575 generic.go:356] "Generic (PLEG): container finished" podID="afda0b32-96d2-430b-ae84-afb262a74141" containerID="83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165" exitCode=0 Jul 20 13:56:40.832887 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.832691 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" Jul 20 13:56:40.832887 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.832718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerDied","Data":"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165"} Jul 20 13:56:40.832887 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.832772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f" event={"ID":"afda0b32-96d2-430b-ae84-afb262a74141","Type":"ContainerDied","Data":"31851c88092acba73c6bb7f0cf267155bee87bb3cbf9e50f4461d883523a5a14"} Jul 20 13:56:40.832887 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.832796 2575 scope.go:117] "RemoveContainer" containerID="83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165" Jul 20 13:56:40.842291 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.842273 2575 scope.go:117] "RemoveContainer" containerID="1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9" Jul 20 13:56:40.851590 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.851561 2575 scope.go:117] "RemoveContainer" containerID="83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165" Jul 20 13:56:40.852096 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:56:40.852072 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165\": container with ID starting with 83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165 not found: ID does not exist" containerID="83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165" Jul 20 13:56:40.852207 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.852105 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165"} err="failed to get container status \"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165\": rpc error: code = NotFound desc = could not find container \"83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165\": container with ID starting with 83d754c42e4e05481a219b0ad68e7305b545123fed66789b5c04cd1a60825165 not found: ID does not exist" Jul 20 13:56:40.852207 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.852125 2575 scope.go:117] "RemoveContainer" containerID="1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9" Jul 20 13:56:40.852435 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:56:40.852411 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9\": container with ID starting with 1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9 not found: ID does not exist" containerID="1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9" Jul 20 13:56:40.852498 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.852442 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9"} err="failed to get container status \"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9\": rpc error: code = NotFound desc = could not find container \"1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9\": container with ID starting with 1baaf55c4769b7288d101dd09a1948ff0074225809563b68756c28d731c2f4d9 not found: ID does not exist" Jul 20 13:56:40.856766 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.856738 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:56:40.862150 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:40.862123 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a45cad8-openshift-default-d958f7b55-2f82f"] Jul 20 13:56:41.882138 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:56:41.882103 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afda0b32-96d2-430b-ae84-afb262a74141" path="/var/lib/kubelet/pods/afda0b32-96d2-430b-ae84-afb262a74141/volumes" Jul 20 13:57:31.237630 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.237542 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 13:57:31.238201 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238121 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:31.238201 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238141 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:31.238201 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238170 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4999954-5d08-4bf1-badd-ee8b2e533a34" containerName="main" Jul 20 13:57:31.238201 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238180 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4999954-5d08-4bf1-badd-ee8b2e533a34" containerName="main" Jul 20 13:57:31.238481 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238273 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4999954-5d08-4bf1-badd-ee8b2e533a34" containerName="main" Jul 20 13:57:31.238481 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238290 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:31.238481 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.238300 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:31.241383 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.241345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.244425 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.244403 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-06fdf5-openshift-default-dockercfg-bj9gw\"" Jul 20 13:57:31.254507 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.254481 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 13:57:31.327298 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327298 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7t59\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327584 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327866 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.327866 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.327739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428626 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7t59\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.428836 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.428900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429130 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.429094 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429344 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.429236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429431 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.429349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429673 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.429648 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.429780 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.429684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.431710 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.431688 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.432015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.431989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.437650 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.437619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.438253 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.438222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7t59\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59\") pod \"e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.553187 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.553071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:31.694996 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.694962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 13:57:31.697123 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:57:31.697095 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c089234_74c8_49cf_8143_c9444b129d95.slice/crio-0bdaa686861e6a5d33e7ff717210704896129d561cef7c2345746d6215324b0f WatchSource:0}: Error finding container 0bdaa686861e6a5d33e7ff717210704896129d561cef7c2345746d6215324b0f: Status 404 returned error can't find the container with id 0bdaa686861e6a5d33e7ff717210704896129d561cef7c2345746d6215324b0f Jul 20 13:57:31.699641 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.699608 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:31.699715 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.699675 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:31.699715 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:31.699706 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:32.077442 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:32.077398 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerStarted","Data":"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88"} Jul 20 13:57:32.077632 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:32.077460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerStarted","Data":"0bdaa686861e6a5d33e7ff717210704896129d561cef7c2345746d6215324b0f"} Jul 20 13:57:32.104967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:32.104915 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podStartSLOduration=1.104898976 podStartE2EDuration="1.104898976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:57:32.102595036 +0000 UTC m=+2450.768534081" watchObservedRunningTime="2026-07-20 13:57:32.104898976 +0000 UTC m=+2450.770838007" Jul 20 13:57:32.553752 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:32.553712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:32.558649 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:32.558620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:33.083026 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:33.082941 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:33.083889 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:33.083869 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:48.342090 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:48.342049 2575 patch_prober.go:28] interesting pod/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:37234->10.133.0.45:15021: read: connection reset by peer" start-of-body= Jul 20 13:57:48.342616 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:48.342139 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:37234->10.133.0.45:15021: read: connection reset by peer" Jul 20 13:57:49.161250 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.161219 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c089234-74c8-49cf-8143-c9444b129d95" containerID="8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88" exitCode=0 Jul 20 13:57:49.161472 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.161300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerDied","Data":"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88"} Jul 20 13:57:49.161813 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.161796 2575 scope.go:117] "RemoveContainer" containerID="8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88" Jul 20 13:57:49.162814 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.162785 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:49.162913 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.162868 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:49.162913 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:49.162901 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:57:50.178600 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:50.178550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerStarted","Data":"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043"} Jul 20 13:57:50.554266 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:50.554160 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:50.554553 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:50.554218 2575 patch_prober.go:28] interesting pod/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 13:57:50.554705 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:50.554682 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 13:57:51.553818 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:51.553775 2575 patch_prober.go:28] interesting pod/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 13:57:51.554380 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:51.553847 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 13:57:52.554388 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:52.554329 2575 patch_prober.go:28] interesting pod/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 13:57:52.555001 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:52.554968 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 13:57:53.556974 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:53.556932 2575 patch_prober.go:28] interesting pod/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 13:57:53.557527 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:53.557020 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 13:57:54.557440 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.557412 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:54.557821 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.557754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:54.558417 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.558397 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 13:57:54.653443 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.653406 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:57:54.653835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.653817 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:54.653835 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.653835 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="afda0b32-96d2-430b-ae84-afb262a74141" containerName="istio-proxy" Jul 20 13:57:54.663843 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.663820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.666644 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.666613 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06fdf5\"/\"openshift-service-ca.crt\"" Jul 20 13:57:54.666793 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.666698 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06fdf5\"/\"default-dockercfg-sskg6\"" Jul 20 13:57:54.666793 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.666613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06fdf5\"/\"test-model-06fdf5-kserve-self-signed-certs\"" Jul 20 13:57:54.667084 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.666915 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06fdf5\"/\"kube-root-ca.crt\"" Jul 20 13:57:54.667084 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.667020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:57:54.768521 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.768711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kl4\" (UniqueName: \"kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.768711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.768711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.768711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.768711 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.768689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.869932 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.869839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kl4\" (UniqueName: \"kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.869932 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.869888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.869936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.869961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.869996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870171 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.870129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870500 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.870344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870500 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.870443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.870597 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.870516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.872201 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.872184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.872625 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.872610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.878635 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.878614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kl4\" (UniqueName: \"kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4\") pod \"test-model-06fdf5-kserve-7c6fcbcddc-lwnrt\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:54.975257 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:54.975217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:57:55.108234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:55.108209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:57:55.109867 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:57:55.109840 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd28110a_f05b_468b_a4ae_f655b1f252ce.slice/crio-89eb89ae5491b56381dc02e11e4252a903487c5e692b494f0b840f4b815de421 WatchSource:0}: Error finding container 89eb89ae5491b56381dc02e11e4252a903487c5e692b494f0b840f4b815de421: Status 404 returned error can't find the container with id 89eb89ae5491b56381dc02e11e4252a903487c5e692b494f0b840f4b815de421 Jul 20 13:57:55.204662 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:55.204625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" event={"ID":"fd28110a-f05b-468b-a4ae-f655b1f252ce","Type":"ContainerStarted","Data":"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312"} Jul 20 13:57:55.204856 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:55.204672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" event={"ID":"fd28110a-f05b-468b-a4ae-f655b1f252ce","Type":"ContainerStarted","Data":"89eb89ae5491b56381dc02e11e4252a903487c5e692b494f0b840f4b815de421"} Jul 20 13:57:55.223446 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:57:55.223398 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" podStartSLOduration=1.223382701 podStartE2EDuration="1.223382701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:57:55.222493716 +0000 UTC m=+2473.888432741" watchObservedRunningTime="2026-07-20 13:57:55.223382701 +0000 UTC m=+2473.889321731" Jul 20 13:58:04.976262 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:04.976213 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:58:04.976262 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:04.976271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:58:04.989056 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:04.989025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:58:05.263949 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:05.263861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:58:36.462213 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.462122 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:58:36.464646 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.464629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.468703 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.468681 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1af9e1\"/\"kube-root-ca.crt\"" Jul 20 13:58:36.468816 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.468683 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1af9e1\"/\"test-model-1af9e1-kserve-self-signed-certs\"" Jul 20 13:58:36.468816 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.468682 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1af9e1\"/\"openshift-service-ca.crt\"" Jul 20 13:58:36.468913 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.468735 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1af9e1\"/\"default-dockercfg-6xjn6\"" Jul 20 13:58:36.475487 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.475462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:58:36.559427 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvpz\" (UniqueName: \"kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.559609 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.559609 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.559609 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.559757 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.559757 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.559647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660757 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvpz\" (UniqueName: \"kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.660967 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.660911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.661289 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.661266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.661343 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.661322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.661404 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.661371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.663256 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.663239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.663366 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.663338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.669096 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.669075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvpz\" (UniqueName: \"kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz\") pod \"test-model-1af9e1-kserve-7dcbdccfbc-b5m4w\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.775573 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.775481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:36.913504 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:36.913480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:58:36.916285 ip-10-0-142-97 kubenswrapper[2575]: W0720 13:58:36.916253 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4a2f556_1ab7_4d14_a31a_8d2c2c57f389.slice/crio-c0f34e9051f4e15240110bb811c13a5e6ff4018507fe0ec02c1c6926c76679d2 WatchSource:0}: Error finding container c0f34e9051f4e15240110bb811c13a5e6ff4018507fe0ec02c1c6926c76679d2: Status 404 returned error can't find the container with id c0f34e9051f4e15240110bb811c13a5e6ff4018507fe0ec02c1c6926c76679d2 Jul 20 13:58:37.417288 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:37.417243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" event={"ID":"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389","Type":"ContainerStarted","Data":"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37"} Jul 20 13:58:37.417288 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:37.417290 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" event={"ID":"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389","Type":"ContainerStarted","Data":"c0f34e9051f4e15240110bb811c13a5e6ff4018507fe0ec02c1c6926c76679d2"} Jul 20 13:58:37.435612 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:37.435557 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" podStartSLOduration=1.435543282 podStartE2EDuration="1.435543282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:58:37.433885718 +0000 UTC m=+2516.099824762" watchObservedRunningTime="2026-07-20 13:58:37.435543282 +0000 UTC m=+2516.101482313" Jul 20 13:58:46.776167 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:46.776121 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:46.776167 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:46.776177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:46.789116 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:46.789085 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:58:47.484637 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:58:47.484606 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:59:32.950225 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:32.950187 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:59:33.887120 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:33.887092 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:59:47.970790 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:47.970744 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" podUID="fd28110a-f05b-468b-a4ae-f655b1f252ce" containerName="main" containerID="cri-o://34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312" gracePeriod=45 Jul 20 13:59:48.219280 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.219247 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:59:48.285463 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285463 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285445 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285499 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2kl4\" (UniqueName: \"kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285529 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285559 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir\") pod \"fd28110a-f05b-468b-a4ae-f655b1f252ce\" (UID: \"fd28110a-f05b-468b-a4ae-f655b1f252ce\") " Jul 20 13:59:48.285702 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285660 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:48.285947 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285810 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home" (OuterVolumeSpecName: "home") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:48.285947 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285853 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:48.286035 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285969 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.286035 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285986 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.286035 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.285998 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.287771 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.287749 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:59:48.288071 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.288048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm" (OuterVolumeSpecName: "dshm") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:48.288145 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.288051 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4" (OuterVolumeSpecName: "kube-api-access-h2kl4") pod "fd28110a-f05b-468b-a4ae-f655b1f252ce" (UID: "fd28110a-f05b-468b-a4ae-f655b1f252ce"). InnerVolumeSpecName "kube-api-access-h2kl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:59:48.386592 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.386550 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd28110a-f05b-468b-a4ae-f655b1f252ce-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.386592 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.386585 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd28110a-f05b-468b-a4ae-f655b1f252ce-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.386592 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.386600 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kl4\" (UniqueName: \"kubernetes.io/projected/fd28110a-f05b-468b-a4ae-f655b1f252ce-kube-api-access-h2kl4\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:48.770619 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.770582 2575 generic.go:356] "Generic (PLEG): container finished" podID="fd28110a-f05b-468b-a4ae-f655b1f252ce" containerID="34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312" exitCode=0 Jul 20 13:59:48.770796 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.770625 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" Jul 20 13:59:48.770796 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.770644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" event={"ID":"fd28110a-f05b-468b-a4ae-f655b1f252ce","Type":"ContainerDied","Data":"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312"} Jul 20 13:59:48.770796 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.770676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt" event={"ID":"fd28110a-f05b-468b-a4ae-f655b1f252ce","Type":"ContainerDied","Data":"89eb89ae5491b56381dc02e11e4252a903487c5e692b494f0b840f4b815de421"} Jul 20 13:59:48.770796 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.770692 2575 scope.go:117] "RemoveContainer" containerID="34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312" Jul 20 13:59:48.780742 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.780713 2575 scope.go:117] "RemoveContainer" containerID="34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312" Jul 20 13:59:48.781018 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:59:48.781000 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312\": container with ID starting with 34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312 not found: ID does not exist" containerID="34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312" Jul 20 13:59:48.781068 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.781026 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312"} err="failed to get container status \"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312\": rpc error: code = NotFound desc = could not find container \"34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312\": container with ID starting with 34b27f6231aa0004b7619ed0d060f0db6b84cfcc86bf48ed0b1655da396ed312 not found: ID does not exist" Jul 20 13:59:48.792398 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.792369 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:59:48.795886 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.795862 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-06fdf5/test-model-06fdf5-kserve-7c6fcbcddc-lwnrt"] Jul 20 13:59:48.907449 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:48.907412 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" podUID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" containerName="main" containerID="cri-o://ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37" gracePeriod=45 Jul 20 13:59:49.166572 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.166546 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:59:49.193921 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.193890 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194083 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.193947 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194083 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194036 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194083 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmvpz\" (UniqueName: \"kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194269 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194120 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194269 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache\") pod \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\" (UID: \"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389\") " Jul 20 13:59:49.194391 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194336 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:49.194450 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home" (OuterVolumeSpecName: "home") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:49.194574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194547 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.194574 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194574 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.194771 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.194555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:49.196234 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.196212 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm" (OuterVolumeSpecName: "dshm") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 13:59:49.196515 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.196495 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:59:49.196515 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.196503 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz" (OuterVolumeSpecName: "kube-api-access-jmvpz") pod "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" (UID: "c4a2f556-1ab7-4d14-a31a-8d2c2c57f389"). InnerVolumeSpecName "kube-api-access-jmvpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:59:49.295721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.295626 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvpz\" (UniqueName: \"kubernetes.io/projected/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-kube-api-access-jmvpz\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.295721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.295665 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.295721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.295674 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.295721 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.295683 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 13:59:49.777015 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.776981 2575 generic.go:356] "Generic (PLEG): container finished" podID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" containerID="ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37" exitCode=0 Jul 20 13:59:49.777217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.777038 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" Jul 20 13:59:49.777217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.777074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" event={"ID":"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389","Type":"ContainerDied","Data":"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37"} Jul 20 13:59:49.777217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.777119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w" event={"ID":"c4a2f556-1ab7-4d14-a31a-8d2c2c57f389","Type":"ContainerDied","Data":"c0f34e9051f4e15240110bb811c13a5e6ff4018507fe0ec02c1c6926c76679d2"} Jul 20 13:59:49.777217 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.777136 2575 scope.go:117] "RemoveContainer" containerID="ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37" Jul 20 13:59:49.786531 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.786510 2575 scope.go:117] "RemoveContainer" containerID="ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37" Jul 20 13:59:49.786794 ip-10-0-142-97 kubenswrapper[2575]: E0720 13:59:49.786776 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37\": container with ID starting with ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37 not found: ID does not exist" containerID="ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37" Jul 20 13:59:49.786842 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.786802 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37"} err="failed to get container status \"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37\": rpc error: code = NotFound desc = could not find container \"ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37\": container with ID starting with ac81d4e3c31f6fe94ac0e6a6c1b9e99424e6bbabf9b1bd18d8f52dc16d7f2a37 not found: ID does not exist" Jul 20 13:59:49.801758 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.801723 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:59:49.805005 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.804977 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1af9e1/test-model-1af9e1-kserve-7dcbdccfbc-b5m4w"] Jul 20 13:59:49.881976 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.881944 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" path="/var/lib/kubelet/pods/c4a2f556-1ab7-4d14-a31a-8d2c2c57f389/volumes" Jul 20 13:59:49.882418 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:49.882400 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd28110a-f05b-468b-a4ae-f655b1f252ce" path="/var/lib/kubelet/pods/fd28110a-f05b-468b-a4ae-f655b1f252ce/volumes" Jul 20 13:59:55.903320 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:55.903287 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 13:59:55.903807 ip-10-0-142-97 kubenswrapper[2575]: I0720 13:59:55.903505 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" containerID="cri-o://86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043" gracePeriod=30 Jul 20 14:00:01.161416 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.161387 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 14:00:01.208974 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.208884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.208974 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.208968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209031 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209064 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7t59\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209119 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209163 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket\") pod \"7c089234-74c8-49cf-8143-c9444b129d95\" (UID: \"7c089234-74c8-49cf-8143-c9444b129d95\") " Jul 20 14:00:01.209214 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209200 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:01.209659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209470 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.209659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209560 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:00:01.209659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209652 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:01.209868 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209816 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:01.210000 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.209886 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:01.211532 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.211506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:00:01.211777 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.211754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:00:01.211777 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.211759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:01.212054 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.212030 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59" (OuterVolumeSpecName: "kube-api-access-b7t59") pod "7c089234-74c8-49cf-8143-c9444b129d95" (UID: "7c089234-74c8-49cf-8143-c9444b129d95"). InnerVolumeSpecName "kube-api-access-b7t59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:01.310798 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310747 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7t59\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-kube-api-access-b7t59\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.310798 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310794 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c089234-74c8-49cf-8143-c9444b129d95-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.310798 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310804 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.310798 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310811 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.310798 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310820 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.311107 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310828 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c089234-74c8-49cf-8143-c9444b129d95-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.311107 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310835 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c089234-74c8-49cf-8143-c9444b129d95-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.311107 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.310843 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c089234-74c8-49cf-8143-c9444b129d95-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:00:01.839273 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.839240 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c089234-74c8-49cf-8143-c9444b129d95" containerID="86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043" exitCode=0 Jul 20 14:00:01.839513 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.839284 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" Jul 20 14:00:01.839513 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.839293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerDied","Data":"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043"} Jul 20 14:00:01.839513 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.839341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx" event={"ID":"7c089234-74c8-49cf-8143-c9444b129d95","Type":"ContainerDied","Data":"0bdaa686861e6a5d33e7ff717210704896129d561cef7c2345746d6215324b0f"} Jul 20 14:00:01.839513 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.839381 2575 scope.go:117] "RemoveContainer" containerID="86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043" Jul 20 14:00:01.849179 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.849162 2575 scope.go:117] "RemoveContainer" containerID="8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88" Jul 20 14:00:01.857976 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.857951 2575 scope.go:117] "RemoveContainer" containerID="86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043" Jul 20 14:00:01.858261 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:00:01.858237 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043\": container with ID starting with 86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043 not found: ID does not exist" containerID="86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043" Jul 20 14:00:01.858320 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.858271 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043"} err="failed to get container status \"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043\": rpc error: code = NotFound desc = could not find container \"86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043\": container with ID starting with 86af3f5f92bb7bec960bf261f29a7459c90fc89d233d82c734b502fe36a6b043 not found: ID does not exist" Jul 20 14:00:01.858320 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.858289 2575 scope.go:117] "RemoveContainer" containerID="8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88" Jul 20 14:00:01.858597 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:00:01.858575 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88\": container with ID starting with 8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88 not found: ID does not exist" containerID="8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88" Jul 20 14:00:01.858698 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.858600 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88"} err="failed to get container status \"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88\": rpc error: code = NotFound desc = could not find container \"8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88\": container with ID starting with 8781731efc0419f05b4ef9b75255836aa25327a7eafe9076cbe6c8a694c1ab88 not found: ID does not exist" Jul 20 14:00:01.870766 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.870733 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 14:00:01.882130 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:01.882105 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-06fdf5-openshift-default-7d5955884c-vwxhx"] Jul 20 14:00:03.881613 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:03.881533 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c089234-74c8-49cf-8143-c9444b129d95" path="/var/lib/kubelet/pods/7c089234-74c8-49cf-8143-c9444b129d95/volumes" Jul 20 14:00:19.205630 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.205597 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.205963 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd28110a-f05b-468b-a4ae-f655b1f252ce" containerName="main" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.205974 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd28110a-f05b-468b-a4ae-f655b1f252ce" containerName="main" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.205995 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206001 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206006 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" containerName="main" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206012 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" containerName="main" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206019 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206024 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206083 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd28110a-f05b-468b-a4ae-f655b1f252ce" containerName="main" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206093 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206101 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c089234-74c8-49cf-8143-c9444b129d95" containerName="istio-proxy" Jul 20 14:00:19.206494 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.206108 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a2f556-1ab7-4d14-a31a-8d2c2c57f389" containerName="main" Jul 20 14:00:19.208965 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.208948 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.211612 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.211585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0f1ced-openshift-default-dockercfg-6kpzg\"" Jul 20 14:00:19.220245 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.220222 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:00:19.380747 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380747 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380998 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380998 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380998 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gv8\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380998 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.380998 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.380953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.381183 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.381010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.381183 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.381028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.481750 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.481750 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.481750 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gv8\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482046 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.481951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482397 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.482152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482397 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.482317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482480 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.482417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482480 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.482444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.482678 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.482656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.484416 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.484388 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.484672 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.484655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.492069 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.492036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.492344 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.492308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gv8\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8\") pod \"e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.519907 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.519882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:19.666693 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:00:19.666661 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472ce5a8_31af_4de9_959d_96decf96c677.slice/crio-441f2a52bf29d8c6a6f21571e3288c72df2222e80c20b74a9a3e9116bb30be7b WatchSource:0}: Error finding container 441f2a52bf29d8c6a6f21571e3288c72df2222e80c20b74a9a3e9116bb30be7b: Status 404 returned error can't find the container with id 441f2a52bf29d8c6a6f21571e3288c72df2222e80c20b74a9a3e9116bb30be7b Jul 20 14:00:19.667226 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.667197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:00:19.669155 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.669118 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:19.669244 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.669201 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:19.669284 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.669244 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:19.932185 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.932153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerStarted","Data":"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b"} Jul 20 14:00:19.932431 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.932215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerStarted","Data":"441f2a52bf29d8c6a6f21571e3288c72df2222e80c20b74a9a3e9116bb30be7b"} Jul 20 14:00:19.959615 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:19.959547 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podStartSLOduration=0.959529752 podStartE2EDuration="959.529752ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:19.957459494 +0000 UTC m=+2618.623398526" watchObservedRunningTime="2026-07-20 14:00:19.959529752 +0000 UTC m=+2618.625468782" Jul 20 14:00:20.520656 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:20.520612 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:20.525814 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:20.525790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:20.936762 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:20.936730 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:20.937814 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:20.937790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:36.978561 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:36.978515 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 14:00:36.978999 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:36.978601 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 14:00:38.024580 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.024539 2575 generic.go:356] "Generic (PLEG): container finished" podID="472ce5a8-31af-4de9-959d-96decf96c677" containerID="239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b" exitCode=0 Jul 20 14:00:38.025009 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.024628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerDied","Data":"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b"} Jul 20 14:00:38.025227 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.025209 2575 scope.go:117] "RemoveContainer" containerID="239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b" Jul 20 14:00:38.026582 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.026554 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:38.026676 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.026631 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:38.026676 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:38.026660 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:00:39.033382 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:39.033320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerStarted","Data":"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e"} Jul 20 14:00:39.520530 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:39.520496 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:39.520758 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:39.520734 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 14:00:39.520847 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:39.520790 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 14:00:40.520526 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:40.520487 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 14:00:40.520905 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:40.520555 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 14:00:41.521337 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:41.521298 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 14:00:41.521795 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:41.521383 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 14:00:42.521171 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:42.521136 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 14:00:42.521372 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:42.521189 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 14:00:43.523909 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:43.523877 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:43.524311 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:43.524247 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:00:43.524910 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:00:43.524892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:01:09.881294 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:09.881264 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:01:09.881809 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:09.881512 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" containerID="cri-o://656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e" gracePeriod=30 Jul 20 14:01:13.526993 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:13.526954 2575 patch_prober.go:28] interesting pod/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 14:01:13.527412 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:13.527030 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 14:01:15.133647 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.133620 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:01:15.206840 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206758 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.206840 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.206840 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206836 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206871 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206905 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.206973 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207158 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96gv8\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8\") pod \"472ce5a8-31af-4de9-959d-96decf96c677\" (UID: \"472ce5a8-31af-4de9-959d-96decf96c677\") " Jul 20 14:01:15.207507 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207200 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:15.207507 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:15.207507 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207390 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:15.207507 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207477 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.207794 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207496 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.207794 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207539 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.207794 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207666 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:01:15.207956 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.207811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data" (OuterVolumeSpecName: "istio-data") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:15.209187 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.209164 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token" (OuterVolumeSpecName: "istio-token") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:01:15.209318 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.209299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:01:15.209565 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.209538 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:01:15.209661 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.209562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8" (OuterVolumeSpecName: "kube-api-access-96gv8") pod "472ce5a8-31af-4de9-959d-96decf96c677" (UID: "472ce5a8-31af-4de9-959d-96decf96c677"). InnerVolumeSpecName "kube-api-access-96gv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:01:15.229145 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.229119 2575 generic.go:356] "Generic (PLEG): container finished" podID="472ce5a8-31af-4de9-959d-96decf96c677" containerID="656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e" exitCode=0 Jul 20 14:01:15.229272 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.229164 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" Jul 20 14:01:15.229272 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.229200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerDied","Data":"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e"} Jul 20 14:01:15.229272 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.229256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc" event={"ID":"472ce5a8-31af-4de9-959d-96decf96c677","Type":"ContainerDied","Data":"441f2a52bf29d8c6a6f21571e3288c72df2222e80c20b74a9a3e9116bb30be7b"} Jul 20 14:01:15.229401 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.229280 2575 scope.go:117] "RemoveContainer" containerID="656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e" Jul 20 14:01:15.238701 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.238680 2575 scope.go:117] "RemoveContainer" containerID="239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b" Jul 20 14:01:15.247078 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.247061 2575 scope.go:117] "RemoveContainer" containerID="656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e" Jul 20 14:01:15.247314 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:01:15.247296 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e\": container with ID starting with 656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e not found: ID does not exist" containerID="656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e" Jul 20 14:01:15.247384 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.247322 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e"} err="failed to get container status \"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e\": rpc error: code = NotFound desc = could not find container \"656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e\": container with ID starting with 656fe8561c19834d6c2ebc43e15d3dfdb08c4a2c36f4d69b92b2fca42b27ce3e not found: ID does not exist" Jul 20 14:01:15.247384 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.247338 2575 scope.go:117] "RemoveContainer" containerID="239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b" Jul 20 14:01:15.247632 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:01:15.247608 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b\": container with ID starting with 239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b not found: ID does not exist" containerID="239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b" Jul 20 14:01:15.247720 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.247635 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b"} err="failed to get container status \"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b\": rpc error: code = NotFound desc = could not find container \"239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b\": container with ID starting with 239ceba83018f191ac20544347fd5d50850b6b4262fa7ac3243183d9f804e84b not found: ID does not exist" Jul 20 14:01:15.254903 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.254882 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:01:15.262031 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.262009 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0f1ced-openshift-default-6cb8f9bbdf-kjwtc"] Jul 20 14:01:15.308536 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308502 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472ce5a8-31af-4de9-959d-96decf96c677-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.308536 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308531 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96gv8\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-kube-api-access-96gv8\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.308536 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308542 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.308763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308554 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472ce5a8-31af-4de9-959d-96decf96c677-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.308763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308561 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472ce5a8-31af-4de9-959d-96decf96c677-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.308763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.308569 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472ce5a8-31af-4de9-959d-96decf96c677-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:01:15.883403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:15.883372 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472ce5a8-31af-4de9-959d-96decf96c677" path="/var/lib/kubelet/pods/472ce5a8-31af-4de9-959d-96decf96c677/volumes" Jul 20 14:01:39.030421 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.030326 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:01:39.030888 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.030869 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.030977 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.030893 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.030977 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.030923 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.030977 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.030932 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.031133 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.031045 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.031133 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.031058 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="472ce5a8-31af-4de9-959d-96decf96c677" containerName="istio-proxy" Jul 20 14:01:39.033395 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.033374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.037804 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.037785 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-001aab-openshift-default-dockercfg-ptl2s\"" Jul 20 14:01:39.055384 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.055340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:01:39.138135 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j58w\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138313 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138313 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138313 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138464 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138464 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138464 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138572 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.138572 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.138493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.239864 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.239826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j58w\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.239864 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.239868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.239896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.239933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.239986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240496 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240496 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240608 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240785 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.240851 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.240810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.241031 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.241006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.242451 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.242429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.242704 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.242684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.255736 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.255716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.257848 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.257832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j58w\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w\") pod \"e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.344102 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.344012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:39.492486 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.492458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:01:39.494410 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:01:39.494381 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42d4d85_2fc6_42c4_9d19_b8fbaeee9313.slice/crio-5c41b181074a089c37e916f91971be39c2334a793402ce2dfe8338cf1e09f2b1 WatchSource:0}: Error finding container 5c41b181074a089c37e916f91971be39c2334a793402ce2dfe8338cf1e09f2b1: Status 404 returned error can't find the container with id 5c41b181074a089c37e916f91971be39c2334a793402ce2dfe8338cf1e09f2b1 Jul 20 14:01:39.496432 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.496398 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:39.496515 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.496462 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:39.496515 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:39.496490 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:40.348912 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:40.348868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerStarted","Data":"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3"} Jul 20 14:01:40.349328 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:40.348935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerStarted","Data":"5c41b181074a089c37e916f91971be39c2334a793402ce2dfe8338cf1e09f2b1"} Jul 20 14:01:40.376710 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:40.376659 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podStartSLOduration=2.376643082 podStartE2EDuration="2.376643082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:01:40.375828329 +0000 UTC m=+2699.041767361" watchObservedRunningTime="2026-07-20 14:01:40.376643082 +0000 UTC m=+2699.042582112" Jul 20 14:01:41.345115 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:41.345074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:41.350139 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:41.350106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:41.354078 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:41.354053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:41.354937 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:41.354915 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:56.565997 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:56.565347 2575 patch_prober.go:28] interesting pod/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:35282->10.133.0.49:15021: read: connection reset by peer" start-of-body= Jul 20 14:01:56.565997 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:56.565433 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:35282->10.133.0.49:15021: read: connection reset by peer" Jul 20 14:01:57.437576 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.437544 2575 generic.go:356] "Generic (PLEG): container finished" podID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerID="65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3" exitCode=0 Jul 20 14:01:57.437752 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.437615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerDied","Data":"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3"} Jul 20 14:01:57.438036 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.438021 2575 scope.go:117] "RemoveContainer" containerID="65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3" Jul 20 14:01:57.439017 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.438988 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:57.439130 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.439056 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:57.439130 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:57.439098 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 14:01:58.446845 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:58.446802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerStarted","Data":"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80"} Jul 20 14:01:59.359381 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:59.356775 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:01:59.359381 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:59.356992 2575 patch_prober.go:28] interesting pod/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 14:01:59.359381 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:01:59.357045 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 14:02:00.345208 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:00.345165 2575 patch_prober.go:28] interesting pod/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 14:02:00.345838 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:00.345801 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 14:02:01.344722 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:01.344675 2575 patch_prober.go:28] interesting pod/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 14:02:01.344940 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:01.344748 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 14:02:02.345060 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:02.345018 2575 patch_prober.go:28] interesting pod/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 14:02:02.345488 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:02.345123 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 14:02:03.348743 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:03.348711 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:02:03.349137 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:03.348999 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:02:03.349861 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:03.349843 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:02:10.566658 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.566624 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:02:10.569183 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.569166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.574666 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.574641 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"kube-root-ca.crt\"" Jul 20 14:02:10.574821 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.574640 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"default-dockercfg-7cn7l\"" Jul 20 14:02:10.574821 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.574696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"auth-test-model-001aab-kserve-self-signed-certs\"" Jul 20 14:02:10.574821 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.574706 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"openshift-service-ca.crt\"" Jul 20 14:02:10.619147 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.619114 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:02:10.682925 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.682890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.682925 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.682929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.683134 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.682949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46dh\" (UniqueName: \"kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.683134 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.683013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.683134 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.683069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.683134 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.683091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.784660 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.784660 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.784892 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.784892 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.784892 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46dh\" (UniqueName: \"kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.785041 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.784909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.785262 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.785238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.785347 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.785264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.785347 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.785287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.787123 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.787096 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.787276 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.787256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.801611 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.801589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46dh\" (UniqueName: \"kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh\") pod \"auth-test-model-001aab-kserve-fdf8fbd67-glwss\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:10.878084 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:10.878063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:11.045315 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:11.045288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:02:11.046537 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:02:11.046511 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98b3806e_b32a_44cc_90d2_bd971c681998.slice/crio-e0298158162b61f8a4fa9de6cffa300da65e80c6dfce840a890d948cc6792b8d WatchSource:0}: Error finding container e0298158162b61f8a4fa9de6cffa300da65e80c6dfce840a890d948cc6792b8d: Status 404 returned error can't find the container with id e0298158162b61f8a4fa9de6cffa300da65e80c6dfce840a890d948cc6792b8d Jul 20 14:02:11.529425 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:11.529385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" event={"ID":"98b3806e-b32a-44cc-90d2-bd971c681998","Type":"ContainerStarted","Data":"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0"} Jul 20 14:02:11.529599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:11.529435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" event={"ID":"98b3806e-b32a-44cc-90d2-bd971c681998","Type":"ContainerStarted","Data":"e0298158162b61f8a4fa9de6cffa300da65e80c6dfce840a890d948cc6792b8d"} Jul 20 14:02:11.560378 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:11.560308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" podStartSLOduration=1.560291361 podStartE2EDuration="1.560291361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:02:11.559865789 +0000 UTC m=+2730.225804821" watchObservedRunningTime="2026-07-20 14:02:11.560291361 +0000 UTC m=+2730.226230392" Jul 20 14:02:20.878177 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:20.878142 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:20.878720 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:20.878350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:20.891369 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:20.891325 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:21.588608 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:21.588569 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:02:38.182144 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.182109 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:02:38.184716 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.184698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.188018 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.187997 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"default-dockercfg-wvdmv\"" Jul 20 14:02:38.189211 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.189192 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"auth-test-model-92659e-kserve-self-signed-certs\"" Jul 20 14:02:38.189321 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.189214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"openshift-service-ca.crt\"" Jul 20 14:02:38.189321 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.189216 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"kube-root-ca.crt\"" Jul 20 14:02:38.201675 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.201648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:02:38.352408 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.352408 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.352659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj22\" (UniqueName: \"kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.352659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.352659 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.352768 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.352656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453712 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453712 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453936 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453936 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj22\" (UniqueName: \"kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453936 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.453936 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.453861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.454193 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.454168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.454287 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.454224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.454379 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.454300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.455987 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.455959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.456290 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.456269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.462904 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.462885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj22\" (UniqueName: \"kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22\") pod \"auth-test-model-92659e-kserve-5865d9fb4c-cmsqk\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.495146 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.495107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:38.651226 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:38.651192 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:02:38.653942 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:02:38.653913 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e92fa0_bd8e_490c_9bc7_265250ea8953.slice/crio-f60103668ecb69400f658815d638fef0fa3446a3f80bf58a7ae515b8b52087f5 WatchSource:0}: Error finding container f60103668ecb69400f658815d638fef0fa3446a3f80bf58a7ae515b8b52087f5: Status 404 returned error can't find the container with id f60103668ecb69400f658815d638fef0fa3446a3f80bf58a7ae515b8b52087f5 Jul 20 14:02:39.666682 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:39.666644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" event={"ID":"d8e92fa0-bd8e-490c-9bc7-265250ea8953","Type":"ContainerStarted","Data":"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84"} Jul 20 14:02:39.666682 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:39.666686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" event={"ID":"d8e92fa0-bd8e-490c-9bc7-265250ea8953","Type":"ContainerStarted","Data":"f60103668ecb69400f658815d638fef0fa3446a3f80bf58a7ae515b8b52087f5"} Jul 20 14:02:39.690481 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:39.690430 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" podStartSLOduration=1.690414444 podStartE2EDuration="1.690414444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:02:39.68992969 +0000 UTC m=+2758.355868721" watchObservedRunningTime="2026-07-20 14:02:39.690414444 +0000 UTC m=+2758.356353474" Jul 20 14:02:48.496221 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:48.496190 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:48.496638 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:48.496238 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:48.508579 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:48.508552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:02:48.720008 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:02:48.719976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:04:41.916013 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:41.915935 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:04:42.771783 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:42.771748 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:04:56.936308 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:56.936262 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" podUID="98b3806e-b32a-44cc-90d2-bd971c681998" containerName="main" containerID="cri-o://49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0" gracePeriod=45 Jul 20 14:04:57.183547 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.183523 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:04:57.197990 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.197922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q46dh\" (UniqueName: \"kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.197990 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.197961 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.197990 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.197981 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.198204 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.198204 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198115 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.198204 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache\") pod \"98b3806e-b32a-44cc-90d2-bd971c681998\" (UID: \"98b3806e-b32a-44cc-90d2-bd971c681998\") " Jul 20 14:04:57.198383 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198216 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home" (OuterVolumeSpecName: "home") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:57.198383 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:57.198495 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198465 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache" (OuterVolumeSpecName: "model-cache") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:57.198610 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198580 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.198610 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198604 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.198720 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.198618 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.200296 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.200267 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm" (OuterVolumeSpecName: "dshm") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:57.200416 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.200397 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh" (OuterVolumeSpecName: "kube-api-access-q46dh") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "kube-api-access-q46dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:04:57.200477 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.200442 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98b3806e-b32a-44cc-90d2-bd971c681998" (UID: "98b3806e-b32a-44cc-90d2-bd971c681998"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:04:57.299958 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.299916 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b3806e-b32a-44cc-90d2-bd971c681998-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.299958 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.299951 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b3806e-b32a-44cc-90d2-bd971c681998-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.299958 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.299966 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46dh\" (UniqueName: \"kubernetes.io/projected/98b3806e-b32a-44cc-90d2-bd971c681998-kube-api-access-q46dh\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:57.368131 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.368093 2575 generic.go:356] "Generic (PLEG): container finished" podID="98b3806e-b32a-44cc-90d2-bd971c681998" containerID="49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0" exitCode=0 Jul 20 14:04:57.368298 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.368143 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" Jul 20 14:04:57.368298 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.368184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" event={"ID":"98b3806e-b32a-44cc-90d2-bd971c681998","Type":"ContainerDied","Data":"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0"} Jul 20 14:04:57.368298 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.368228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss" event={"ID":"98b3806e-b32a-44cc-90d2-bd971c681998","Type":"ContainerDied","Data":"e0298158162b61f8a4fa9de6cffa300da65e80c6dfce840a890d948cc6792b8d"} Jul 20 14:04:57.368298 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.368251 2575 scope.go:117] "RemoveContainer" containerID="49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0" Jul 20 14:04:57.377872 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.377846 2575 scope.go:117] "RemoveContainer" containerID="49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0" Jul 20 14:04:57.378135 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:04:57.378117 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0\": container with ID starting with 49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0 not found: ID does not exist" containerID="49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0" Jul 20 14:04:57.378219 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.378143 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0"} err="failed to get container status \"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0\": rpc error: code = NotFound desc = could not find container \"49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0\": container with ID starting with 49010d918dabf6a832717cf60ee43a9a7936dcf10c03c2e80a4846f764c8fff0 not found: ID does not exist" Jul 20 14:04:57.397865 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.397827 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:04:57.401322 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.401297 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/auth-test-model-001aab-kserve-fdf8fbd67-glwss"] Jul 20 14:04:57.792017 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.791979 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" podUID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" containerName="main" containerID="cri-o://d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84" gracePeriod=45 Jul 20 14:04:57.882182 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:57.882152 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b3806e-b32a-44cc-90d2-bd971c681998" path="/var/lib/kubelet/pods/98b3806e-b32a-44cc-90d2-bd971c681998/volumes" Jul 20 14:04:58.055420 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.055344 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:04:58.107400 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107400 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107667 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107449 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107667 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107468 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107667 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107524 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdj22\" (UniqueName: \"kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107667 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm\") pod \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\" (UID: \"d8e92fa0-bd8e-490c-9bc7-265250ea8953\") " Jul 20 14:04:58.107873 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107642 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:58.107873 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107734 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home" (OuterVolumeSpecName: "home") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:58.107873 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:58.108004 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107879 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.108004 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107898 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.108004 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.107912 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.109817 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.109795 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm" (OuterVolumeSpecName: "dshm") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:04:58.109923 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.109828 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22" (OuterVolumeSpecName: "kube-api-access-vdj22") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "kube-api-access-vdj22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:04:58.109923 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.109905 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8e92fa0-bd8e-490c-9bc7-265250ea8953" (UID: "d8e92fa0-bd8e-490c-9bc7-265250ea8953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:04:58.208538 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.208496 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e92fa0-bd8e-490c-9bc7-265250ea8953-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.208538 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.208529 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdj22\" (UniqueName: \"kubernetes.io/projected/d8e92fa0-bd8e-490c-9bc7-265250ea8953-kube-api-access-vdj22\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.208538 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.208538 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e92fa0-bd8e-490c-9bc7-265250ea8953-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:04:58.373855 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.373776 2575 generic.go:356] "Generic (PLEG): container finished" podID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" containerID="d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84" exitCode=0 Jul 20 14:04:58.373855 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.373817 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" Jul 20 14:04:58.374077 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.373862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" event={"ID":"d8e92fa0-bd8e-490c-9bc7-265250ea8953","Type":"ContainerDied","Data":"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84"} Jul 20 14:04:58.374077 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.373916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk" event={"ID":"d8e92fa0-bd8e-490c-9bc7-265250ea8953","Type":"ContainerDied","Data":"f60103668ecb69400f658815d638fef0fa3446a3f80bf58a7ae515b8b52087f5"} Jul 20 14:04:58.374077 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.373943 2575 scope.go:117] "RemoveContainer" containerID="d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84" Jul 20 14:04:58.382962 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.382945 2575 scope.go:117] "RemoveContainer" containerID="d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84" Jul 20 14:04:58.383196 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:04:58.383180 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84\": container with ID starting with d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84 not found: ID does not exist" containerID="d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84" Jul 20 14:04:58.383269 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.383244 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84"} err="failed to get container status \"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84\": rpc error: code = NotFound desc = could not find container \"d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84\": container with ID starting with d39732509f893c19a4d04573769decafcefa8279198881251e1c8c057cc28e84 not found: ID does not exist" Jul 20 14:04:58.395496 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.395465 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:04:58.401368 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:58.401334 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/auth-test-model-92659e-kserve-5865d9fb4c-cmsqk"] Jul 20 14:04:59.882295 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:04:59.882262 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" path="/var/lib/kubelet/pods/d8e92fa0-bd8e-490c-9bc7-265250ea8953/volumes" Jul 20 14:05:15.333419 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333384 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333753 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b3806e-b32a-44cc-90d2-bd971c681998" containerName="main" Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333763 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b3806e-b32a-44cc-90d2-bd971c681998" containerName="main" Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333774 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" containerName="main" Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333781 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" containerName="main" Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333848 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b3806e-b32a-44cc-90d2-bd971c681998" containerName="main" Jul 20 14:05:15.333883 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.333856 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e92fa0-bd8e-490c-9bc7-265250ea8953" containerName="main" Jul 20 14:05:15.338056 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.338039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.344615 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.344328 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"kube-root-ca.crt\"" Jul 20 14:05:15.344615 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.344328 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"default-dockercfg-wvdmv\"" Jul 20 14:05:15.344615 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.344334 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"sub-test-model-92659e-kserve-self-signed-certs\"" Jul 20 14:05:15.344781 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.344740 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"openshift-service-ca.crt\"" Jul 20 14:05:15.352346 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.352321 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:05:15.477278 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.477278 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.477588 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.477588 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.477588 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdk4\" (UniqueName: \"kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.477588 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.477539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578520 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578520 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578789 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578789 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578789 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdk4\" (UniqueName: \"kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.578789 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.579057 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.578989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.579057 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.579014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.579057 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.579028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.581026 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.580997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.581302 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.581282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.587693 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.587633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdk4\" (UniqueName: \"kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4\") pod \"sub-test-model-92659e-kserve-7bd478c799-2fbm9\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.647338 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.647298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:15.793132 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:15.788895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:05:16.465826 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:16.465788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" event={"ID":"01e97ae8-ee57-4fd8-bb7a-79c786d707d1","Type":"ContainerStarted","Data":"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe"} Jul 20 14:05:16.465826 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:16.465831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" event={"ID":"01e97ae8-ee57-4fd8-bb7a-79c786d707d1","Type":"ContainerStarted","Data":"e273d87501fb480cbf23d0330a9357a7d58641bdcb46c0a89a6a674ee76e74de"} Jul 20 14:05:16.484825 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:16.484754 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" podStartSLOduration=1.484733185 podStartE2EDuration="1.484733185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:05:16.484222118 +0000 UTC m=+2915.150161149" watchObservedRunningTime="2026-07-20 14:05:16.484733185 +0000 UTC m=+2915.150672217" Jul 20 14:05:25.647783 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:25.647739 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:25.647783 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:25.647785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:25.660216 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:25.660186 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:26.527281 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:26.527250 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:05:56.940185 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:56.940150 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:05:59.295855 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.295819 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:05:59.299749 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.299718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.304628 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.304587 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"openshift-service-ca.crt\"" Jul 20 14:05:59.304787 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.304636 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"default-dockercfg-7cn7l\"" Jul 20 14:05:59.304787 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.304636 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"rate-test-model-001aab-kserve-self-signed-certs\"" Jul 20 14:05:59.304787 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.304602 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"kube-root-ca.crt\"" Jul 20 14:05:59.316810 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.316782 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:05:59.395781 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.395781 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.396043 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.396043 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.396043 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.396043 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.395926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hkb\" (UniqueName: \"kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.496851 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.496818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497055 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.496865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497055 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.496985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hkb\" (UniqueName: \"kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497178 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497178 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497178 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497339 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497339 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.497528 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.497507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.499679 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.499656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.500334 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.500306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.509957 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.509935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hkb\" (UniqueName: \"kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb\") pod \"rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.609262 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.609174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:05:59.755679 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:05:59.755654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:05:59.757987 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:05:59.757959 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaddd517a_c337_436c_aeec_485d8868acf8.slice/crio-eb6508084d2ddc9ff27b8c13358cde028e4ca7139bd30f0c154b7856141d008f WatchSource:0}: Error finding container eb6508084d2ddc9ff27b8c13358cde028e4ca7139bd30f0c154b7856141d008f: Status 404 returned error can't find the container with id eb6508084d2ddc9ff27b8c13358cde028e4ca7139bd30f0c154b7856141d008f Jul 20 14:06:00.682884 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:00.682850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" event={"ID":"addd517a-c337-436c-aeec-485d8868acf8","Type":"ContainerStarted","Data":"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e"} Jul 20 14:06:00.682884 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:00.682890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" event={"ID":"addd517a-c337-436c-aeec-485d8868acf8","Type":"ContainerStarted","Data":"eb6508084d2ddc9ff27b8c13358cde028e4ca7139bd30f0c154b7856141d008f"} Jul 20 14:06:00.703667 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:00.703618 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" podStartSLOduration=1.7036024950000002 podStartE2EDuration="1.703602495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:06:00.702898585 +0000 UTC m=+2959.368837616" watchObservedRunningTime="2026-07-20 14:06:00.703602495 +0000 UTC m=+2959.369541525" Jul 20 14:06:09.609701 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:09.609600 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:06:09.609701 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:09.609661 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:06:09.621843 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:09.621816 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:06:09.739197 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:09.739163 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:06:11.961021 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:11.960970 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" podUID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" containerName="main" containerID="cri-o://e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe" gracePeriod=45 Jul 20 14:06:12.198058 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.198032 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:06:12.321200 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321104 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kdk4\" (UniqueName: \"kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321200 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321157 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321200 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321184 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321468 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321389 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321468 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:12.321586 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321586 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321470 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home" (OuterVolumeSpecName: "home") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:12.321586 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs\") pod \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\" (UID: \"01e97ae8-ee57-4fd8-bb7a-79c786d707d1\") " Jul 20 14:06:12.321741 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321611 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:12.321932 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321916 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.322007 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321937 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.322007 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.321950 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.323495 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.323470 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4" (OuterVolumeSpecName: "kube-api-access-8kdk4") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "kube-api-access-8kdk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:06:12.323886 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.323866 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:06:12.323950 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.323888 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm" (OuterVolumeSpecName: "dshm") pod "01e97ae8-ee57-4fd8-bb7a-79c786d707d1" (UID: "01e97ae8-ee57-4fd8-bb7a-79c786d707d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:12.423221 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.423181 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.423221 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.423215 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kdk4\" (UniqueName: \"kubernetes.io/projected/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-kube-api-access-8kdk4\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.423221 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.423228 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e97ae8-ee57-4fd8-bb7a-79c786d707d1-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.743556 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.743521 2575 generic.go:356] "Generic (PLEG): container finished" podID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" containerID="e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe" exitCode=0 Jul 20 14:06:12.743763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.743567 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" Jul 20 14:06:12.743763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.743614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" event={"ID":"01e97ae8-ee57-4fd8-bb7a-79c786d707d1","Type":"ContainerDied","Data":"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe"} Jul 20 14:06:12.743763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.743666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9" event={"ID":"01e97ae8-ee57-4fd8-bb7a-79c786d707d1","Type":"ContainerDied","Data":"e273d87501fb480cbf23d0330a9357a7d58641bdcb46c0a89a6a674ee76e74de"} Jul 20 14:06:12.743763 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.743689 2575 scope.go:117] "RemoveContainer" containerID="e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe" Jul 20 14:06:12.753184 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.753165 2575 scope.go:117] "RemoveContainer" containerID="e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe" Jul 20 14:06:12.753470 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:06:12.753446 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe\": container with ID starting with e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe not found: ID does not exist" containerID="e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe" Jul 20 14:06:12.753531 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.753480 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe"} err="failed to get container status \"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe\": rpc error: code = NotFound desc = could not find container \"e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe\": container with ID starting with e8e6f137a0b26d76591a2ad31e72185866effe3e74800df47ef0aea999c8b6fe not found: ID does not exist" Jul 20 14:06:12.765603 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.765575 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:06:12.771740 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:12.771716 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/sub-test-model-92659e-kserve-7bd478c799-2fbm9"] Jul 20 14:06:13.885795 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:13.885764 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" path="/var/lib/kubelet/pods/01e97ae8-ee57-4fd8-bb7a-79c786d707d1/volumes" Jul 20 14:06:20.164133 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.164096 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:06:20.164577 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.164561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" containerName="main" Jul 20 14:06:20.164626 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.164580 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" containerName="main" Jul 20 14:06:20.164670 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.164661 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e97ae8-ee57-4fd8-bb7a-79c786d707d1" containerName="main" Jul 20 14:06:20.169155 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.169133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.173420 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.173395 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"rate-test-model-92659e-kserve-self-signed-certs\"" Jul 20 14:06:20.173420 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.173400 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"openshift-service-ca.crt\"" Jul 20 14:06:20.173950 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.173934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"default-dockercfg-wvdmv\"" Jul 20 14:06:20.174191 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.174179 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-92659e\"/\"kube-root-ca.crt\"" Jul 20 14:06:20.181520 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.181497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:06:20.207771 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.207738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.207976 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.207795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.207976 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.207853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.207976 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.207895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.207976 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.207930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkv9p\" (UniqueName: \"kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.208115 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.208016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.308854 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.308820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309032 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.308873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309032 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.308893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309032 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.308914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309032 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.308954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkv9p\" (UniqueName: \"kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309032 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.309028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309289 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.309230 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309345 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.309327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.309432 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.309394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.311252 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.311229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.311529 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.311511 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.321105 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.321082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkv9p\" (UniqueName: \"kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p\") pod \"rate-test-model-92659e-kserve-54644bd775-zb5ck\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.479346 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.479251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:20.615222 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.615198 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:06:20.617227 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:06:20.617201 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a74a014_c3d9_4a9e_b44e_ef670c55e371.slice/crio-b1e1cae0353154161d582b0d212fd50b9dbe32b0bcf89cc6bfd41824242cb8f3 WatchSource:0}: Error finding container b1e1cae0353154161d582b0d212fd50b9dbe32b0bcf89cc6bfd41824242cb8f3: Status 404 returned error can't find the container with id b1e1cae0353154161d582b0d212fd50b9dbe32b0bcf89cc6bfd41824242cb8f3 Jul 20 14:06:20.784452 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.784331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" event={"ID":"0a74a014-c3d9-4a9e-b44e-ef670c55e371","Type":"ContainerStarted","Data":"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108"} Jul 20 14:06:20.784452 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.784410 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" event={"ID":"0a74a014-c3d9-4a9e-b44e-ef670c55e371","Type":"ContainerStarted","Data":"b1e1cae0353154161d582b0d212fd50b9dbe32b0bcf89cc6bfd41824242cb8f3"} Jul 20 14:06:20.821144 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:20.821091 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" podStartSLOduration=0.821075643 podStartE2EDuration="821.075643ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:06:20.819559384 +0000 UTC m=+2979.485498414" watchObservedRunningTime="2026-07-20 14:06:20.821075643 +0000 UTC m=+2979.487014674" Jul 20 14:06:30.479767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:30.479718 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:30.479767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:30.479781 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:30.492099 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:30.492066 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:06:30.845637 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:06:30.845549 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:07:00.756928 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:00.756892 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:07:01.425105 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:01.425071 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:07:15.776955 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:15.776904 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" podUID="addd517a-c337-436c-aeec-485d8868acf8" containerName="main" containerID="cri-o://b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e" gracePeriod=45 Jul 20 14:07:16.029920 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.029863 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:07:16.068364 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.068324 2575 generic.go:356] "Generic (PLEG): container finished" podID="addd517a-c337-436c-aeec-485d8868acf8" containerID="b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e" exitCode=0 Jul 20 14:07:16.068543 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.068388 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" Jul 20 14:07:16.068543 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.068399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" event={"ID":"addd517a-c337-436c-aeec-485d8868acf8","Type":"ContainerDied","Data":"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e"} Jul 20 14:07:16.068543 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.068453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb" event={"ID":"addd517a-c337-436c-aeec-485d8868acf8","Type":"ContainerDied","Data":"eb6508084d2ddc9ff27b8c13358cde028e4ca7139bd30f0c154b7856141d008f"} Jul 20 14:07:16.068543 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.068477 2575 scope.go:117] "RemoveContainer" containerID="b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e" Jul 20 14:07:16.077098 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.077083 2575 scope.go:117] "RemoveContainer" containerID="b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e" Jul 20 14:07:16.077344 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:07:16.077328 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e\": container with ID starting with b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e not found: ID does not exist" containerID="b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e" Jul 20 14:07:16.077418 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.077368 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e"} err="failed to get container status \"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e\": rpc error: code = NotFound desc = could not find container \"b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e\": container with ID starting with b7f23ff8104477ef31e6d88cf3f93fd9d868e57c5b9d57411d46ded17c4d2a7e not found: ID does not exist" Jul 20 14:07:16.163422 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163389 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163606 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163446 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163606 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163471 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163606 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163515 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163606 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163547 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163820 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163608 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7hkb\" (UniqueName: \"kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb\") pod \"addd517a-c337-436c-aeec-485d8868acf8\" (UID: \"addd517a-c337-436c-aeec-485d8868acf8\") " Jul 20 14:07:16.163820 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163714 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.163820 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163779 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache" (OuterVolumeSpecName: "model-cache") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.163820 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.163788 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home" (OuterVolumeSpecName: "home") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.164026 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.164009 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.164069 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.164030 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.164069 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.164045 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.165803 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.165779 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm" (OuterVolumeSpecName: "dshm") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.165928 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.165810 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:07:16.165928 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.165889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb" (OuterVolumeSpecName: "kube-api-access-x7hkb") pod "addd517a-c337-436c-aeec-485d8868acf8" (UID: "addd517a-c337-436c-aeec-485d8868acf8"). InnerVolumeSpecName "kube-api-access-x7hkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:16.265174 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.265142 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hkb\" (UniqueName: \"kubernetes.io/projected/addd517a-c337-436c-aeec-485d8868acf8-kube-api-access-x7hkb\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.265174 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.265167 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/addd517a-c337-436c-aeec-485d8868acf8-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.265174 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.265181 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/addd517a-c337-436c-aeec-485d8868acf8-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.392013 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.391972 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:07:16.396827 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.396801 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/rate-test-model-001aab-kserve-cf6f4b8c5-ntjxb"] Jul 20 14:07:16.445629 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.445584 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" podUID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" containerName="main" containerID="cri-o://4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108" gracePeriod=45 Jul 20 14:07:16.696874 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.696847 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:07:16.870834 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.870781 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.870834 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.870846 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.870874 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.870948 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.870979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkv9p\" (UniqueName: \"kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871005 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache\") pod \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\" (UID: \"0a74a014-c3d9-4a9e-b44e-ef670c55e371\") " Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871092 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home" (OuterVolumeSpecName: "home") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871132 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.871403 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871306 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.871791 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871436 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-home\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.871791 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871449 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tmp-dir\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.871791 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.871457 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-model-cache\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.873118 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.873092 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:07:16.873233 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.873131 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm" (OuterVolumeSpecName: "dshm") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:16.873233 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.873156 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p" (OuterVolumeSpecName: "kube-api-access-zkv9p") pod "0a74a014-c3d9-4a9e-b44e-ef670c55e371" (UID: "0a74a014-c3d9-4a9e-b44e-ef670c55e371"). InnerVolumeSpecName "kube-api-access-zkv9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:16.972761 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.972665 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a74a014-c3d9-4a9e-b44e-ef670c55e371-dshm\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.972761 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.972695 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a74a014-c3d9-4a9e-b44e-ef670c55e371-tls-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:16.972761 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:16.972705 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkv9p\" (UniqueName: \"kubernetes.io/projected/0a74a014-c3d9-4a9e-b44e-ef670c55e371-kube-api-access-zkv9p\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:17.075471 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.075434 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" containerID="4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108" exitCode=0 Jul 20 14:07:17.075654 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.075478 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" Jul 20 14:07:17.075654 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.075511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" event={"ID":"0a74a014-c3d9-4a9e-b44e-ef670c55e371","Type":"ContainerDied","Data":"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108"} Jul 20 14:07:17.075654 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.075550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck" event={"ID":"0a74a014-c3d9-4a9e-b44e-ef670c55e371","Type":"ContainerDied","Data":"b1e1cae0353154161d582b0d212fd50b9dbe32b0bcf89cc6bfd41824242cb8f3"} Jul 20 14:07:17.075654 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.075567 2575 scope.go:117] "RemoveContainer" containerID="4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108" Jul 20 14:07:17.085018 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.084996 2575 scope.go:117] "RemoveContainer" containerID="4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108" Jul 20 14:07:17.085311 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:07:17.085292 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108\": container with ID starting with 4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108 not found: ID does not exist" containerID="4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108" Jul 20 14:07:17.085394 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.085321 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108"} err="failed to get container status \"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108\": rpc error: code = NotFound desc = could not find container \"4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108\": container with ID starting with 4b30228a5dda10768480b13022cba1605df2bf9f8a87555e40d049858b1ed108 not found: ID does not exist" Jul 20 14:07:17.103125 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.103093 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:07:17.106931 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.106904 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-92659e/rate-test-model-92659e-kserve-54644bd775-zb5ck"] Jul 20 14:07:17.881677 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.881642 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" path="/var/lib/kubelet/pods/0a74a014-c3d9-4a9e-b44e-ef670c55e371/volumes" Jul 20 14:07:17.882098 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:17.882081 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="addd517a-c337-436c-aeec-485d8868acf8" path="/var/lib/kubelet/pods/addd517a-c337-436c-aeec-485d8868acf8/volumes" Jul 20 14:07:42.766198 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:42.766150 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:07:42.766734 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:42.766507 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" containerID="cri-o://df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80" gracePeriod=30 Jul 20 14:07:48.016007 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.015980 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:07:48.042779 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042699 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.042779 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.042779 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042754 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j58w\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042802 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042912 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.042947 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.043013 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.043039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket\") pod \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\" (UID: \"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313\") " Jul 20 14:07:48.043657 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.043157 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:48.043657 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.043409 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-certs\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.043657 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.043634 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:48.044228 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.044189 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data" (OuterVolumeSpecName: "istio-data") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:48.044463 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.044421 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:48.044575 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.044464 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:07:48.045610 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.045587 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 14:07:48.045840 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.045822 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token" (OuterVolumeSpecName: "istio-token") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:48.045912 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.045840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w" (OuterVolumeSpecName: "kube-api-access-2j58w") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "kube-api-access-2j58w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:48.046205 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.046188 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" (UID: "b42d4d85-2fc6-42c4-9d19-b8fbaeee9313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:07:48.144194 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144152 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-podinfo\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144194 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144184 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-envoy\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144194 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144195 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-credential-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144194 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144204 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-workload-socket\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144508 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144213 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istiod-ca-cert\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144508 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144221 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-token\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144508 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144230 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j58w\" (UniqueName: \"kubernetes.io/projected/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-kube-api-access-2j58w\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.144508 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.144237 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313-istio-data\") on node \"ip-10-0-142-97.ec2.internal\" DevicePath \"\"" Jul 20 14:07:48.224586 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.224548 2575 generic.go:356] "Generic (PLEG): container finished" podID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerID="df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80" exitCode=0 Jul 20 14:07:48.224788 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.224601 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" Jul 20 14:07:48.224788 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.224634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerDied","Data":"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80"} Jul 20 14:07:48.224788 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.224705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s" event={"ID":"b42d4d85-2fc6-42c4-9d19-b8fbaeee9313","Type":"ContainerDied","Data":"5c41b181074a089c37e916f91971be39c2334a793402ce2dfe8338cf1e09f2b1"} Jul 20 14:07:48.224788 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.224738 2575 scope.go:117] "RemoveContainer" containerID="df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80" Jul 20 14:07:48.234153 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.234135 2575 scope.go:117] "RemoveContainer" containerID="65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3" Jul 20 14:07:48.242429 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.242408 2575 scope.go:117] "RemoveContainer" containerID="df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80" Jul 20 14:07:48.242741 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:07:48.242718 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80\": container with ID starting with df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80 not found: ID does not exist" containerID="df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80" Jul 20 14:07:48.242803 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.242751 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80"} err="failed to get container status \"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80\": rpc error: code = NotFound desc = could not find container \"df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80\": container with ID starting with df472d43accfe7f37222c142b0c5716b509ac814c74ecc5ca85a25cc66c15b80 not found: ID does not exist" Jul 20 14:07:48.242803 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.242769 2575 scope.go:117] "RemoveContainer" containerID="65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3" Jul 20 14:07:48.243017 ip-10-0-142-97 kubenswrapper[2575]: E0720 14:07:48.242997 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3\": container with ID starting with 65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3 not found: ID does not exist" containerID="65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3" Jul 20 14:07:48.243063 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.243024 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3"} err="failed to get container status \"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3\": rpc error: code = NotFound desc = could not find container \"65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3\": container with ID starting with 65bd2f20568d98d7023b045e06aac93c09d3eeaaaa1f3dd956864236624838f3 not found: ID does not exist" Jul 20 14:07:48.247145 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.247111 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:07:48.251605 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:48.251581 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-001aab-openshift-default-7559cff59f-84c9s"] Jul 20 14:07:49.881523 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:07:49.881488 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" path="/var/lib/kubelet/pods/b42d4d85-2fc6-42c4-9d19-b8fbaeee9313/volumes" Jul 20 14:08:06.310711 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:06.310678 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd4b9b4f4-9xlcm_7a27a718-1c81-4cc0-80e0-b38ea84e9be4/authorino/0.log" Jul 20 14:08:12.558589 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:12.558555 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nkdnv_9ea63735-8420-4324-8fc7-d90dea2bf75c/server/0.log" Jul 20 14:08:12.800865 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:12.800834 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-hw7xp_4425b323-c82f-4897-a79f-9410096fba6a/copy-manifests/0.log" Jul 20 14:08:12.808328 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:12.808302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-hw7xp_4425b323-c82f-4897-a79f-9410096fba6a/manager/0.log" Jul 20 14:08:15.183964 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:15.183934 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd4b9b4f4-9xlcm_7a27a718-1c81-4cc0-80e0-b38ea84e9be4/authorino/0.log" Jul 20 14:08:16.404968 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:16.404931 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm5g9_7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af/istio-proxy/0.log" Jul 20 14:08:16.646038 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:16.646008 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-8w6nl_20b277b7-d465-488a-bf2a-622498290e40/kube-auth-proxy/0.log" Jul 20 14:08:17.284057 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:17.284024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-786b55b76d-4gts5_6a56b5a7-6007-44c7-a47f-1bc6a516826d/router/0.log" Jul 20 14:08:17.779209 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:17.779177 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tt56x_eeb09e15-a5f3-4627-a674-fb8406054db5/main/0.log" Jul 20 14:08:17.899192 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:17.899148 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ktrdm_5fc46f50-70e6-4072-a392-724703838449/main/0.log" Jul 20 14:08:18.259033 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:18.259004 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7w5mm_74a4475b-8f87-41f6-9059-14ef1818f363/main/0.log" Jul 20 14:08:27.464814 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:27.464776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rvdwx_5e8945d1-09f8-44da-aeca-94ff385f58f8/global-pull-secret-syncer/0.log" Jul 20 14:08:27.590183 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:27.590143 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8dgkv_d83c7a14-b7c1-4951-a080-f57d421fd337/konnectivity-agent/0.log" Jul 20 14:08:27.712102 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:27.712058 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-97.ec2.internal_3259959531b2cae4a95970f39ca273a7/haproxy/0.log" Jul 20 14:08:34.741160 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:34.741124 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bd4b9b4f4-9xlcm_7a27a718-1c81-4cc0-80e0-b38ea84e9be4/authorino/0.log" Jul 20 14:08:36.912970 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:36.912940 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/alertmanager/0.log" Jul 20 14:08:36.958591 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:36.958560 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/config-reloader/0.log" Jul 20 14:08:37.047694 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.047658 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/kube-rbac-proxy-web/0.log" Jul 20 14:08:37.065195 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065165 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b"] Jul 20 14:08:37.065548 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065536 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="addd517a-c337-436c-aeec-485d8868acf8" containerName="main" Jul 20 14:08:37.065599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065550 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="addd517a-c337-436c-aeec-485d8868acf8" containerName="main" Jul 20 14:08:37.065599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065562 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" Jul 20 14:08:37.065599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065567 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" Jul 20 14:08:37.065599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065575 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" containerName="main" Jul 20 14:08:37.065599 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065581 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" containerName="main" Jul 20 14:08:37.065755 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065645 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="addd517a-c337-436c-aeec-485d8868acf8" containerName="main" Jul 20 14:08:37.065755 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065653 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42d4d85-2fc6-42c4-9d19-b8fbaeee9313" containerName="istio-proxy" Jul 20 14:08:37.065755 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.065663 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a74a014-c3d9-4a9e-b44e-ef670c55e371" containerName="main" Jul 20 14:08:37.068668 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.068652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.072861 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.072841 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lp56k\"/\"default-dockercfg-m47wx\"" Jul 20 14:08:37.073224 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.073207 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"openshift-service-ca.crt\"" Jul 20 14:08:37.077781 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.077759 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"kube-root-ca.crt\"" Jul 20 14:08:37.077954 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.077935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-podres\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.078057 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.078043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-lib-modules\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.078108 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.078066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cff\" (UniqueName: \"kubernetes.io/projected/e81f77c7-65f7-423e-bfa4-364d2991a4ed-kube-api-access-k4cff\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.078147 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.078114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-proc\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.078182 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.078158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-sys\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.103097 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.103061 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b"] Jul 20 14:08:37.130049 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.130025 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/kube-rbac-proxy/0.log" Jul 20 14:08:37.179544 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-podres\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179544 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-lib-modules\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cff\" (UniqueName: \"kubernetes.io/projected/e81f77c7-65f7-423e-bfa4-364d2991a4ed-kube-api-access-k4cff\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-proc\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-podres\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-sys\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-proc\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-lib-modules\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.179767 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.179730 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e81f77c7-65f7-423e-bfa4-364d2991a4ed-sys\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.180464 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.180445 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/kube-rbac-proxy-metric/0.log" Jul 20 14:08:37.192745 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.192721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cff\" (UniqueName: \"kubernetes.io/projected/e81f77c7-65f7-423e-bfa4-364d2991a4ed-kube-api-access-k4cff\") pod \"perf-node-gather-daemonset-9nb4b\" (UID: \"e81f77c7-65f7-423e-bfa4-364d2991a4ed\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.210186 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.210155 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/prom-label-proxy/0.log" Jul 20 14:08:37.260331 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.260295 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4042774-51a5-4116-9ed0-c9bb3bf2f710/init-config-reloader/0.log" Jul 20 14:08:37.378231 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.378190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:37.525331 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.525305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b"] Jul 20 14:08:37.527273 ip-10-0-142-97 kubenswrapper[2575]: W0720 14:08:37.527241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode81f77c7_65f7_423e_bfa4_364d2991a4ed.slice/crio-dfe414f4637011cf432453d0e695e594d902a5dcc703891629afc48560925d3c WatchSource:0}: Error finding container dfe414f4637011cf432453d0e695e594d902a5dcc703891629afc48560925d3c: Status 404 returned error can't find the container with id dfe414f4637011cf432453d0e695e594d902a5dcc703891629afc48560925d3c Jul 20 14:08:37.905561 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.905534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9b9x_14397bd9-1bec-4120-af3f-40aaed40255b/node-exporter/0.log" Jul 20 14:08:37.935102 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.935055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9b9x_14397bd9-1bec-4120-af3f-40aaed40255b/kube-rbac-proxy/0.log" Jul 20 14:08:37.977563 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:37.977539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x9b9x_14397bd9-1bec-4120-af3f-40aaed40255b/init-textfile/0.log" Jul 20 14:08:38.070220 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.070193 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hshbj_3416af86-934d-430e-88e4-fb8bc068f9f4/kube-rbac-proxy-main/0.log" Jul 20 14:08:38.154744 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.154718 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hshbj_3416af86-934d-430e-88e4-fb8bc068f9f4/kube-rbac-proxy-self/0.log" Jul 20 14:08:38.254594 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.254513 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hshbj_3416af86-934d-430e-88e4-fb8bc068f9f4/openshift-state-metrics/0.log" Jul 20 14:08:38.467870 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.467835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" event={"ID":"e81f77c7-65f7-423e-bfa4-364d2991a4ed","Type":"ContainerStarted","Data":"c7f81fb078df5bb5093417e1345a353ef2a4e6a87ec0bde75779dd0ce496ca20"} Jul 20 14:08:38.468052 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.467891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" event={"ID":"e81f77c7-65f7-423e-bfa4-364d2991a4ed","Type":"ContainerStarted","Data":"dfe414f4637011cf432453d0e695e594d902a5dcc703891629afc48560925d3c"} Jul 20 14:08:38.468052 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.467923 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:38.891002 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.890972 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ff4874b9-ghvd7_67494da6-17a8-4f54-bc5d-d081f86d1407/telemeter-client/0.log" Jul 20 14:08:38.918949 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.918917 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ff4874b9-ghvd7_67494da6-17a8-4f54-bc5d-d081f86d1407/reload/0.log" Jul 20 14:08:38.962556 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:38.962523 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ff4874b9-ghvd7_67494da6-17a8-4f54-bc5d-d081f86d1407/kube-rbac-proxy/0.log" Jul 20 14:08:39.013822 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.013797 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/thanos-query/0.log" Jul 20 14:08:39.053910 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.053886 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/kube-rbac-proxy-web/0.log" Jul 20 14:08:39.089691 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.089661 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/kube-rbac-proxy/0.log" Jul 20 14:08:39.129759 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.129726 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/prom-label-proxy/0.log" Jul 20 14:08:39.181791 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.181700 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/kube-rbac-proxy-rules/0.log" Jul 20 14:08:39.217228 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:39.217202 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb4f6cc59-tqgsp_61359bfc-4ca0-4ffb-ae57-72b567c4c100/kube-rbac-proxy-metrics/0.log" Jul 20 14:08:40.281568 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:40.281529 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-j2tlx_5fb80410-b6cb-4568-aa83-bdea7c63f351/networking-console-plugin/0.log" Jul 20 14:08:41.591274 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:41.591244 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f6d9b5758-54mgr_45aabc49-e711-4d00-878d-0c13f7bfaf60/console/0.log" Jul 20 14:08:43.544037 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:43.543993 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8m4g_c3d5296d-3f5d-4f43-9633-a31bcf70b0ff/dns/0.log" Jul 20 14:08:43.601982 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:43.601950 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8m4g_c3d5296d-3f5d-4f43-9633-a31bcf70b0ff/kube-rbac-proxy/0.log" Jul 20 14:08:43.816253 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:43.816161 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drxgx_f9cb3873-9970-4689-9e1f-423a90921969/dns-node-resolver/0.log" Jul 20 14:08:44.485720 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:44.485694 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" Jul 20 14:08:44.521179 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:44.521127 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-9nb4b" podStartSLOduration=7.521109606 podStartE2EDuration="7.521109606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 14:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:08:38.49598138 +0000 UTC m=+3117.161920413" watchObservedRunningTime="2026-07-20 14:08:44.521109606 +0000 UTC m=+3123.187048649" Jul 20 14:08:44.748006 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:44.747927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stgxm_97144075-89d8-4105-a155-f224c2724c68/node-ca/0.log" Jul 20 14:08:45.873873 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:45.873773 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm5g9_7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af/istio-proxy/0.log" Jul 20 14:08:46.031670 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:46.031638 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-8w6nl_20b277b7-d465-488a-bf2a-622498290e40/kube-auth-proxy/0.log" Jul 20 14:08:46.278699 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:46.278642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-786b55b76d-4gts5_6a56b5a7-6007-44c7-a47f-1bc6a516826d/router/0.log" Jul 20 14:08:47.315197 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.315116 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hnnd_cc55c466-02a5-40b0-bd25-230d7b714fcb/serve-healthcheck-canary/0.log" Jul 20 14:08:47.581543 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.581418 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm5g9_7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af/istio-proxy/0.log" Jul 20 14:08:47.705145 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.705114 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-8w6nl_20b277b7-d465-488a-bf2a-622498290e40/kube-auth-proxy/0.log" Jul 20 14:08:47.876117 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.876026 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-786b55b76d-4gts5_6a56b5a7-6007-44c7-a47f-1bc6a516826d/router/0.log" Jul 20 14:08:47.968515 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.968474 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mvx7_a986f2ae-eac9-4fe7-8f37-c18b44ed11f1/kube-rbac-proxy/0.log" Jul 20 14:08:47.991992 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:47.991959 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mvx7_a986f2ae-eac9-4fe7-8f37-c18b44ed11f1/exporter/0.log" Jul 20 14:08:48.013175 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:48.013142 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mvx7_a986f2ae-eac9-4fe7-8f37-c18b44ed11f1/extractor/0.log" Jul 20 14:08:49.098739 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:49.098713 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm5g9_7c84e28a-d2b9-4aaf-aa4c-1886aea1d0af/istio-proxy/0.log" Jul 20 14:08:49.226397 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:49.226269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c5c8fcf86-8w6nl_20b277b7-d465-488a-bf2a-622498290e40/kube-auth-proxy/0.log" Jul 20 14:08:49.443464 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:49.443409 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-786b55b76d-4gts5_6a56b5a7-6007-44c7-a47f-1bc6a516826d/router/0.log" Jul 20 14:08:51.479577 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:51.479544 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nkdnv_9ea63735-8420-4324-8fc7-d90dea2bf75c/server/0.log" Jul 20 14:08:51.722940 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:51.722909 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-hw7xp_4425b323-c82f-4897-a79f-9410096fba6a/manager/0.log" Jul 20 14:08:51.782662 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:51.782581 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-hw7xp_4425b323-c82f-4897-a79f-9410096fba6a/copy-manifests/0.log" Jul 20 14:08:53.292541 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:53.292506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-548487689c-cn7cm_c5c0ca60-20ae-4f3e-a9ef-f577f8733a0f/manager/0.log" Jul 20 14:08:53.369974 ip-10-0-142-97 kubenswrapper[2575]: I0720 14:08:53.369933 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wjm9c_ee9ba7ae-4763-40b9-b0d5-4ea2deb6ef91/openshift-lws-operator/0.log"