Aug 05 17:20:15.065710 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 05 17:20:15.065721 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 05 17:20:15.065729 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 05 17:20:15.066029 ip-10-0-131-14 systemd[1]: Failed to start Kubernetes Kubelet. Aug 05 17:20:25.308200 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 05 17:20:25.308215 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 033bd9bc821e44df8bf3d1d0e3ab0f27 -- Aug 05 17:22:22.216381 ip-10-0-131-14 systemd[1]: Starting Kubernetes Kubelet... Aug 05 17:22:22.623015 ip-10-0-131-14 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:22:22.623015 ip-10-0-131-14 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 17:22:22.623015 ip-10-0-131-14 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:22:22.623015 ip-10-0-131-14 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 17:22:22.623015 ip-10-0-131-14 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:22:22.624925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.624837 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 17:22:22.629220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629205 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.629220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629221 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629225 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629228 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629231 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629234 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629237 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629239 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629242 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629247 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629251 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629254 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629257 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629259 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629262 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629265 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629267 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629270 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629273 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629276 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.629282 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629279 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629282 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629284 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629288 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629290 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629293 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629296 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629298 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629301 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629303 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629306 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629308 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629310 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629313 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629315 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629319 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629323 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629326 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629329 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629332 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.629777 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629334 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629337 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629339 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629342 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629344 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629347 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629349 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629352 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629355 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629358 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629361 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629364 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629366 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629368 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629372 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629375 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629377 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629380 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629382 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629385 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.630274 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629388 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629390 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629392 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629395 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629398 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629400 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629403 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629405 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629408 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629411 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629413 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629416 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629418 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629422 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629424 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629427 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629429 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629432 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629434 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.630753 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629438 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629443 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629446 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629449 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629452 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629455 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629457 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629461 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629464 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629466 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629469 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629472 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629475 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629477 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629480 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629482 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629485 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629488 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629490 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.631220 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.629493 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630248 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630255 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630260 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630269 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630272 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630275 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630278 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630281 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630284 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630287 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630289 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630292 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630294 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630297 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630300 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630302 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630305 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630307 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630310 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.631682 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630313 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630315 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630318 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630320 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630323 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630326 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630328 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630331 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630333 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630335 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630338 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630341 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630343 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630346 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630348 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630351 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630355 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630363 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630366 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.632187 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630369 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630371 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630373 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630376 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630378 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630381 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630384 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630387 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630389 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630392 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630394 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630397 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630400 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630403 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630406 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630408 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630411 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630413 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630416 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630419 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.632708 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630421 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630424 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630426 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630428 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630431 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630434 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630436 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630439 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630441 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630444 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630446 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630455 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630458 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630460 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630463 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630466 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630469 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630471 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630474 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.633238 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630476 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630478 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630481 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630483 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630486 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630488 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630491 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630493 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630496 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630498 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630501 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630504 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630507 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630510 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630512 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630517 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630519 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630522 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630524 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630527 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.633719 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630529 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630533 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630981 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630986 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630995 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.630998 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631000 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631003 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631005 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631008 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631010 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631013 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631016 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631018 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631021 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631023 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631026 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631029 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631034 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.634223 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631038 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631041 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631044 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631047 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631050 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631053 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631055 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631058 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631062 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631064 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631067 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631069 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631072 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631074 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631077 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631079 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631082 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631084 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631092 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631095 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.634715 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631097 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631100 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631116 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631119 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631121 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631124 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631127 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631131 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631133 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631136 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631139 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631142 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631145 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631149 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631152 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631155 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631157 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631161 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631164 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.635199 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631166 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631171 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631173 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631176 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631178 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631181 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631183 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631186 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631188 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631191 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631193 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631196 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631207 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631210 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631213 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631215 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631218 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631220 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631223 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.635659 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631226 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631228 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631231 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631233 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631236 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631239 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631242 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631244 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631247 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631249 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631252 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631254 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631257 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631259 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631263 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631265 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631268 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631270 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631273 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631275 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.636152 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631278 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631280 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631282 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.631285 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631369 2576 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631379 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631406 2576 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631410 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631414 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631417 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631421 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631425 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631428 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631431 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631434 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631437 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631440 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631444 2576 flags.go:64] FLAG: --cgroup-root="" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631447 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631450 2576 flags.go:64] FLAG: --client-ca-file="" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631452 2576 flags.go:64] FLAG: --cloud-provider="external" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631455 2576 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631462 2576 flags.go:64] FLAG: --cluster-domain="" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631465 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 17:22:22.636636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631468 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631471 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631476 2576 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631480 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631483 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631486 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631489 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631492 2576 flags.go:64] FLAG: --contention-profiling="false" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631495 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631498 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631501 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631504 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631508 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631511 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631514 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631522 2576 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631525 2576 flags.go:64] FLAG: --enable-server="true" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631528 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631535 2576 flags.go:64] FLAG: --event-burst="100" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631538 2576 flags.go:64] FLAG: --event-qps="50" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631541 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631545 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631547 2576 flags.go:64] FLAG: --eviction-hard="" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631551 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 17:22:22.637233 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631554 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631557 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631560 2576 flags.go:64] FLAG: --eviction-soft="" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631563 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631566 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631569 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631572 2576 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631574 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631577 2576 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631580 2576 flags.go:64] FLAG: --feature-gates="" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631585 2576 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631588 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631591 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631594 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631597 2576 flags.go:64] FLAG: --healthz-port="10248" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631600 2576 flags.go:64] FLAG: --help="false" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631603 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-14.ec2.internal" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631606 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631608 2576 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631611 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631615 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631618 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631621 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 17:22:22.638021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631623 2576 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631632 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631635 2576 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631638 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631641 2576 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631644 2576 flags.go:64] FLAG: --kube-reserved="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631646 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631649 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631652 2576 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631655 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631657 2576 flags.go:64] FLAG: --lock-file="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631660 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631663 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631666 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631671 2576 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631673 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631676 2576 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631678 2576 flags.go:64] FLAG: --logging-format="text" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631681 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631686 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631689 2576 flags.go:64] FLAG: --manifest-url="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631692 2576 flags.go:64] FLAG: --manifest-url-header="" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631696 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631699 2576 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631703 2576 flags.go:64] FLAG: --max-pods="110" Aug 05 17:22:22.639030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631706 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631709 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631711 2576 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631714 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631717 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631720 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631723 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631729 2576 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631732 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631742 2576 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631746 2576 flags.go:64] FLAG: --pod-cidr="" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631749 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631754 2576 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631757 2576 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631760 2576 flags.go:64] FLAG: --pods-per-core="0" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631763 2576 flags.go:64] FLAG: --port="10250" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631766 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631769 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-036ee26eca46914f2" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631771 2576 flags.go:64] FLAG: --qos-reserved="" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631774 2576 flags.go:64] FLAG: --read-only-port="10255" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631777 2576 flags.go:64] FLAG: --register-node="true" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631780 2576 flags.go:64] FLAG: --register-with-taints="" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631784 2576 flags.go:64] FLAG: --registry-burst="10" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631787 2576 flags.go:64] FLAG: --registry-qps="5" Aug 05 17:22:22.640073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631790 2576 flags.go:64] FLAG: --reserved-cpus="" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631793 2576 flags.go:64] FLAG: --reserved-memory="" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631797 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631800 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631803 2576 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631806 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631808 2576 flags.go:64] FLAG: --runonce="false" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631811 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631814 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631817 2576 flags.go:64] FLAG: --seccomp-default="false" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631820 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631823 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631826 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631829 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631832 2576 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631834 2576 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631837 2576 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631840 2576 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631849 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631853 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631855 2576 flags.go:64] FLAG: --system-cgroups="" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631858 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631863 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631866 2576 flags.go:64] FLAG: --tls-cert-file="" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631869 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 17:22:22.640991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631875 2576 flags.go:64] FLAG: --tls-min-version="" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631878 2576 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631880 2576 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631883 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631886 2576 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631889 2576 flags.go:64] FLAG: --v="2" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631893 2576 flags.go:64] FLAG: --version="false" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631897 2576 flags.go:64] FLAG: --vmodule="" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631900 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.631904 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632040 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632046 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632050 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632053 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632056 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632059 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632062 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632065 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632068 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632071 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632074 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632077 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.641617 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632079 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632082 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632084 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632089 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632092 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632094 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632097 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632100 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632114 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632116 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632119 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632122 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632125 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632127 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632130 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632133 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632135 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632138 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632140 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.642198 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632143 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632146 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632148 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632151 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632153 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632156 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632159 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632161 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632164 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632166 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632168 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632171 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632174 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632176 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632179 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632181 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632185 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632187 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632191 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632194 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.642643 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632196 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632199 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632201 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632204 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632207 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632209 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632212 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632214 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632217 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632221 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632225 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632227 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632230 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632233 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632235 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632238 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632240 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632243 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632246 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.643142 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632248 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632251 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632253 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632256 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632258 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632261 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632263 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632266 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632268 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632272 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632274 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632277 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632280 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632282 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632285 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632288 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632290 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632293 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632297 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632301 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.643595 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632304 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632307 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632310 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632312 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632315 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632317 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632320 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632322 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.632325 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.632330 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.641201 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.641215 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.642001 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.642019 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642075 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.644085 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642080 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642083 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642087 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642090 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642093 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642095 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642098 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642116 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642122 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642125 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642128 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642130 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642133 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642136 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642138 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642141 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642144 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642147 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642149 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642152 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.644471 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642155 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642157 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642160 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642162 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642165 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642168 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642171 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642174 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642176 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642179 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642182 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642184 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642187 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642190 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642193 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642195 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642198 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642201 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642204 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642208 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.644951 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642210 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642213 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642216 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642218 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642223 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642226 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642229 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642232 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642236 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642238 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642241 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642245 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642248 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642251 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642253 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642256 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642258 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642261 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642264 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642266 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.645438 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642269 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642271 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642274 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642276 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642279 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642281 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642284 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642286 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642290 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642293 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642296 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642299 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642302 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642305 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642308 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642310 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642313 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642315 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642325 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.645927 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642328 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642331 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642334 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642337 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642339 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642342 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642344 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642347 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642349 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642352 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642371 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642375 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642378 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642382 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642385 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642387 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642390 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642393 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.646401 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642395 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.642401 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642512 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642518 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642521 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642524 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642527 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642530 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642533 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642535 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642538 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642541 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642544 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642547 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:22:22.646841 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642555 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642559 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642562 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642565 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642569 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642571 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642574 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642576 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642579 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642581 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642584 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642587 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642589 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642592 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642594 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642597 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642600 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642603 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642606 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:22:22.647266 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642608 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642611 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642613 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642616 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642618 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642621 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642624 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642627 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642629 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642632 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642634 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642637 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642639 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642642 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642650 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642653 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642656 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642659 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642661 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642664 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:22:22.647716 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642666 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642669 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642672 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642674 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642676 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642679 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642681 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642684 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642686 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642689 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642692 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642694 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642697 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642699 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642702 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642704 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642707 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642709 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642711 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642714 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:22:22.648210 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642716 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642719 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642722 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642724 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642726 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642729 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642731 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642739 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642742 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642744 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642747 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642750 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642752 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642755 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642757 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642760 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642764 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642767 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642770 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642772 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:22:22.648692 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642775 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642778 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642780 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642784 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642788 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642790 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642793 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:22.642796 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.642800 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.642903 2576 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.647036 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.648358 2576 server.go:1019] "Starting client certificate rotation" Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.648453 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:22:22.649178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.648488 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:22:22.650886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.650872 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:22:22.652060 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.652043 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:22:22.658193 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.658175 2576 log.go:25] "Validated CRI v1 runtime API" Aug 05 17:22:22.660246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.660231 2576 log.go:25] "Validated CRI v1 image API" Aug 05 17:22:22.661560 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.661545 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 17:22:22.663891 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.663874 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8cda54bb-c062-4634-9963-9ec042743d32:/dev/nvme0n1p3 f9f15699-6247-4268-9ae9-8020bc0524c0:/dev/nvme0n1p4] Aug 05 17:22:22.663947 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:22.663891 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 17:22:23.771642 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.771533 2576 manager.go:216] Machine: {Timestamp:2026-08-05 17:22:23.770022797 +0000 UTC m=+0.331308403 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104298 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec211a71e6f31f70ad8a6f677d813910 SystemUUID:ec211a71-e6f3-1f70-ad8a-6f677d813910 BootID:033bd9bc-821e-44df-8bf3-d1d0e3ab0f27 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:57:df:c0:ae:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:57:df:c0:ae:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:ff:e4:3f:91:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 17:22:23.771642 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.771636 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 17:22:23.771760 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.771710 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 17:22:23.774802 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.774780 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 17:22:23.774934 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.774806 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 17:22:23.775523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.775513 2576 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 17:22:23.775555 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.775525 2576 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 17:22:23.775555 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.775532 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:22:23.776469 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.776458 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:22:23.777742 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.777728 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 17:22:23.779753 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.779738 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:22:23.781303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781291 2576 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:22:23.781514 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781504 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 17:22:23.781547 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781534 2576 kubelet.go:486] "Attempting to sync node with API server" Aug 05 17:22:23.781547 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781543 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 17:22:23.781598 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781559 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 17:22:23.781598 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781569 2576 kubelet.go:398] "Adding apiserver pod source" Aug 05 17:22:23.781598 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781579 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 17:22:23.781957 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781942 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:22:23.782011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.781967 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:22:23.785363 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.785343 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 17:22:23.786702 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.786688 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 17:22:23.786751 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.786718 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 17:22:23.788116 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788104 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788121 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788127 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788132 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788141 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788147 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788154 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788159 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788166 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788171 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 17:22:23.788179 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788180 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 17:22:23.788457 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.788185 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 17:22:23.789041 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.789028 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 17:22:23.789041 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.789038 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 17:22:23.792333 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.792310 2576 server.go:1268] "Started kubelet" Aug 05 17:22:23.792429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.792398 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 17:22:23.792575 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.792534 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 17:22:23.792665 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.792598 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 17:22:23.792940 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.792914 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 17:22:23.793026 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.792978 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 17:22:23.793073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.793033 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 17:22:23.793126 ip-10-0-131-14 systemd[1]: Started Kubernetes Kubelet. Aug 05 17:22:23.793710 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.793689 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 17:22:23.794533 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.794521 2576 server.go:310] "Adding debug handlers to kubelet server" Aug 05 17:22:23.797505 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.797474 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 17:22:23.799554 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.799536 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 17:22:23.799656 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.799564 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 17:22:23.799721 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.799651 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:23.800207 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800189 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 17:22:23.800207 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800192 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 17:22:23.800356 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800216 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 17:22:23.800408 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800371 2576 reconstruct.go:97] "Volume reconstruction finished" Aug 05 17:22:23.800408 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800381 2576 reconciler.go:29] "Reconciler: start to sync state" Aug 05 17:22:23.800728 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.799678 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18c8f88ea527a44a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-08-05 17:22:23.792284746 +0000 UTC m=+0.353570350,LastTimestamp:2026-08-05 17:22:23.792284746 +0000 UTC m=+0.353570350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}" Aug 05 17:22:23.800817 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800764 2576 factory.go:153] Registering CRI-O factory Aug 05 17:22:23.800817 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800777 2576 factory.go:223] Registration of the crio container factory successfully Aug 05 17:22:23.800917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800836 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 17:22:23.800917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800846 2576 factory.go:55] Registering systemd factory Aug 05 17:22:23.800917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800854 2576 factory.go:223] Registration of the systemd container factory successfully Aug 05 17:22:23.800917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800873 2576 factory.go:103] Registering Raw factory Aug 05 17:22:23.800917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.800885 2576 manager.go:1196] Started watching for new ooms in manager Aug 05 17:22:23.801392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.801379 2576 manager.go:349] Starting recovery of all containers Aug 05 17:22:23.807444 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.807420 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 05 17:22:23.808053 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.807801 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 05 17:22:23.809870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.809832 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 17:22:23.814776 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.814755 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvbcd" Aug 05 17:22:23.815291 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.815276 2576 manager.go:354] Recovery completed Aug 05 17:22:23.819236 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.819224 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:23.820475 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.820458 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvbcd" Aug 05 17:22:23.821066 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821050 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:23.821147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821084 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:23.821147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821099 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:23.821625 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821609 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 17:22:23.821625 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821622 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 17:22:23.821734 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821628 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:23.821734 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.821641 2576 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:22:23.822909 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.822896 2576 policy_none.go:49] "None policy: Start" Aug 05 17:22:23.822969 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.822918 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 17:22:23.822969 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.822929 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 17:22:23.823034 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.822893 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:23.823034 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.822997 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:23.823034 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.823009 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:23.823154 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.823093 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18c8f88ea6ded0a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-08-05 17:22:23.821066401 +0000 UTC m=+0.382352009,LastTimestamp:2026-08-05 17:22:23.821066401 +0000 UTC m=+0.382352009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}" Aug 05 17:22:23.824062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.824051 2576 policy_none.go:47] "Start" Aug 05 17:22:23.863007 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.862873 2576 manager.go:337] "Starting Device Plugin manager" Aug 05 17:22:23.863095 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.863039 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 17:22:23.863095 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863052 2576 server.go:87] "Starting device plugin registration server" Aug 05 17:22:23.863381 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863367 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 17:22:23.863434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863383 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 17:22:23.863533 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863516 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 17:22:23.863637 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863626 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 17:22:23.863637 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.863636 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 17:22:23.864075 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.864056 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 17:22:23.864154 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.864108 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:23.963778 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.963752 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:23.965147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.965131 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:23.965211 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.965162 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:23.965211 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.965176 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:23.965211 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.965209 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-14.ec2.internal" Aug 05 17:22:23.971724 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.971704 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 17:22:23.971806 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.971743 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 17:22:23.971806 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.971765 2576 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 17:22:23.971881 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.971794 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 17:22:23.971881 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.971810 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-14.ec2.internal" Aug 05 17:22:23.971881 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:23.971833 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-14.ec2.internal\": node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:23.973952 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:23.973936 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:22:24.013080 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.013034 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.072302 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.072280 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"] Aug 05 17:22:24.072406 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.072362 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:24.073190 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.073176 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:24.073253 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.073205 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:24.073253 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.073214 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:24.074558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.074546 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:24.074687 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.074674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.074725 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.074701 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:24.075213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075195 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:24.075314 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075221 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:24.075314 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075230 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:24.075314 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075195 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:24.075314 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075296 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:24.075314 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.075310 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:24.076532 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.076519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.076575 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.076542 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:22:24.077225 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.077210 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:22:24.077286 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.077239 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:22:24.077286 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.077253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:22:24.101770 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.101750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.101841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.101776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.101841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.101792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c415e6671b04646ed97d73439cdf78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"1c415e6671b04646ed97d73439cdf78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.107042 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.107028 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.111445 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.111429 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.113756 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.113744 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.201983 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.201959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.202070 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.201986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.202070 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.202004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c415e6671b04646ed97d73439cdf78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"1c415e6671b04646ed97d73439cdf78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.202070 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.202045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.202070 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.202064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c415e6671b04646ed97d73439cdf78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"1c415e6671b04646ed97d73439cdf78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.202208 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.202080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/095232a6940bc44f8134e96f40d4f622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"095232a6940bc44f8134e96f40d4f622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.214083 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.214063 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.314869 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.314803 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.409380 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.409354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.414139 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.413802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:24.414880 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.414864 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.515409 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.515367 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.615890 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.615816 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.716377 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.716347 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.750513 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.750487 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 17:22:24.750894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.750634 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 17:22:24.817387 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.817365 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:24.824526 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.824501 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:17:23 +0000 UTC" deadline="2028-01-06 12:24:57.136190003 +0000 UTC" Aug 05 17:22:24.824526 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.824524 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12451h2m32.311669251s" Aug 05 17:22:24.857739 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.857707 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:22:24.859804 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:24.859787 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:22:24.917975 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:24.917954 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Aug 05 17:22:25.008827 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:25.008795 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095232a6940bc44f8134e96f40d4f622.slice/crio-641d8a24407800f228c7e8a8c92b416739a754c8bbd58218dd1841d67b742159 WatchSource:0}: Error finding container 641d8a24407800f228c7e8a8c92b416739a754c8bbd58218dd1841d67b742159: Status 404 returned error can't find the container with id 641d8a24407800f228c7e8a8c92b416739a754c8bbd58218dd1841d67b742159 Aug 05 17:22:25.010147 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:25.010122 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c415e6671b04646ed97d73439cdf78e.slice/crio-d4be6c78dcee543fa535ff21089cb04cbc66dffa3e118ac0efd910291438d5a0 WatchSource:0}: Error finding container d4be6c78dcee543fa535ff21089cb04cbc66dffa3e118ac0efd910291438d5a0: Status 404 returned error can't find the container with id d4be6c78dcee543fa535ff21089cb04cbc66dffa3e118ac0efd910291438d5a0 Aug 05 17:22:25.013858 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.013839 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:22:25.014589 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.014493 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:22:25.100130 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.100099 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Aug 05 17:22:25.113304 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.113282 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:22:25.114480 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.114467 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Aug 05 17:22:25.121028 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.121015 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:22:25.782544 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.782504 2576 apiserver.go:52] "Watching apiserver" Aug 05 17:22:25.789896 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.789874 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 17:22:25.790272 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.790247 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-thxcm","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal","openshift-cluster-node-tuning-operator/tuned-sgc2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","openshift-multus/network-metrics-daemon-6x9n7","openshift-network-operator/iptables-alerter-8sbxs","openshift-ovn-kubernetes/ovnkube-node-9sjc7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj","openshift-dns/node-resolver-qh92q","openshift-image-registry/node-ca-cfvz2","openshift-multus/multus-additional-cni-plugins-zjp4n","openshift-multus/multus-pvqz7","openshift-network-diagnostics/network-check-target-pf69t","kube-system/global-pull-secret-syncer-cc44b"] Aug 05 17:22:25.791849 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.791827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.793603 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.792983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.794427 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.794406 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.794530 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.794442 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.794530 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.794505 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q4lw4\"" Aug 05 17:22:25.794784 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.794761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 17:22:25.794999 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.794983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.795092 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.795076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vq56f\"" Aug 05 17:22:25.795398 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.795082 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.795623 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.795607 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 17:22:25.795687 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.795655 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.796967 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.796943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.797054 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:25.797008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:25.797132 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.797114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 17:22:25.797389 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.797372 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75mpn\"" Aug 05 17:22:25.797925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.797907 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.798117 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.798102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 17:22:25.798356 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.798341 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.798436 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.798403 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 17:22:25.798572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.798554 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 17:22:25.798639 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.798603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.799528 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.799217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.800376 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.800159 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 17:22:25.800649 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.800628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.801159 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801027 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.801251 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801205 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 17:22:25.801251 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6m8rt\"" Aug 05 17:22:25.801385 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801253 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.801385 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801259 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 17:22:25.801570 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2tnh\"" Aug 05 17:22:25.802003 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.801986 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 17:22:25.802315 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.802297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:25.803613 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.803381 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.803903 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.803884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bmfv\"" Aug 05 17:22:25.804355 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.804290 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.807850 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.807830 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.807948 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.807930 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.808008 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.807963 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfx75\"" Aug 05 17:22:25.808384 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.808365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.810399 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.810380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.810968 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.810950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 17:22:25.811085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-conf\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-systemd\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-sys\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811237 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-lib-modules\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811237 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.811237 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhqd\" (UniqueName: \"kubernetes.io/projected/5d60c593-81e3-418c-bbea-0747ab88a80b-kube-api-access-vdhqd\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.811237 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pwx\" (UniqueName: \"kubernetes.io/projected/045408ce-93d2-435e-9f6c-8b9e0e454631-kube-api-access-d9pwx\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.811237 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-slash\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysconfig\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-tmp\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/045408ce-93d2-435e-9f6c-8b9e0e454631-host\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-systemd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811347 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-var-lib-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-etc-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-log-socket\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-kubernetes\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-var-lib-kubelet\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811474 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 17:22:25.811493 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-etc-tuned\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-socket-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-device-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811555 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811690 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2pr4m\"" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-node-log\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-sys-fs\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-kubelet\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-bin\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-script-lib\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmnkz\" (UniqueName: \"kubernetes.io/projected/3a143562-c6dd-444d-b26c-eb6758aa7afd-kube-api-access-kmnkz\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.811945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thl2g\" (UniqueName: \"kubernetes.io/projected/4968ea50-1e0b-49c5-86a5-6c8747e563e9-kube-api-access-thl2g\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm7x\" (UniqueName: \"kubernetes.io/projected/545d6d14-ef1f-41eb-962a-f4eef7fbade3-kube-api-access-lmm7x\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.812068 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-registration-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-systemd-units\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-ovn\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:25.812115 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-config\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-run\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovn-node-metrics-cert\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545d6d14-ef1f-41eb-962a-f4eef7fbade3-iptables-alerter-script\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1e5680e-ed52-40a5-8225-973f315728b2-agent-certs\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1e5680e-ed52-40a5-8225-973f315728b2-konnectivity-ca\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-host\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812214 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6xh9w\"" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n82r\" (UniqueName: \"kubernetes.io/projected/79ed7e8f-7393-470b-acca-337a55a5542b-kube-api-access-8n82r\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/045408ce-93d2-435e-9f6c-8b9e0e454631-serviceca\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.812791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545d6d14-ef1f-41eb-962a-f4eef7fbade3-host-slash\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-modprobe-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-netns\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-netd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-env-overrides\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812875 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.812994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:25.813580 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:25.813043 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:25.840791 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.840768 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gqfhv" Aug 05 17:22:25.846435 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.846415 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gqfhv" Aug 05 17:22:25.901827 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.901776 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 17:22:25.913002 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.912975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545d6d14-ef1f-41eb-962a-f4eef7fbade3-host-slash\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.913002 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-netd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cni-binary-copy\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-netns\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545d6d14-ef1f-41eb-962a-f4eef7fbade3-host-slash\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-netd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-bin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:25.913240 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-kubelet\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-bin\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-script-lib\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-cni-bin\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:25.913501 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:26.413439116 +0000 UTC m=+2.974724712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:25.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmnkz\" (UniqueName: \"kubernetes.io/projected/3a143562-c6dd-444d-b26c-eb6758aa7afd-kube-api-access-kmnkz\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thl2g\" (UniqueName: \"kubernetes.io/projected/4968ea50-1e0b-49c5-86a5-6c8747e563e9-kube-api-access-thl2g\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-conf\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-sys\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-sys\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhqd\" (UniqueName: \"kubernetes.io/projected/5d60c593-81e3-418c-bbea-0747ab88a80b-kube-api-access-vdhqd\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-conf\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/045408ce-93d2-435e-9f6c-8b9e0e454631-host\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-var-lib-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-etc-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/045408ce-93d2-435e-9f6c-8b9e0e454631-host\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.913916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-log-socket\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-etc-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/544f2794-e894-49e8-b7e9-bb2b030fa972-hosts-file\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-var-lib-openvswitch\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-log-socket\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-etc-tuned\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-node-log\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7c8v\" (UniqueName: \"kubernetes.io/projected/544f2794-e894-49e8-b7e9-bb2b030fa972-kube-api-access-v7c8v\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-os-release\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-node-log\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.913997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-hostroot\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-conf-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-daemon-config\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzrh\" (UniqueName: \"kubernetes.io/projected/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-kube-api-access-2hzrh\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-kubelet\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-script-lib\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.914653 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-socket-dir-parent\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914149 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-kubelet\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-registration-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-ovn\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-registration-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-system-cni-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-run\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-kubelet-config\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-multus\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-host\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-ovn\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-run\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-host\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.914890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-env-overrides\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.915421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-modprobe-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-netns\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-run-netns\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-env-overrides\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-modprobe-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-dbus\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-cnibin\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdshz\" (UniqueName: \"kubernetes.io/projected/0035653b-7120-4e6d-b28c-b136ba8bd846-kube-api-access-vdshz\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-multus-certs\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-systemd\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-lib-modules\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.915992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pwx\" (UniqueName: \"kubernetes.io/projected/045408ce-93d2-435e-9f6c-8b9e0e454631-kube-api-access-d9pwx\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-slash\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-systemd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-lib-modules\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-run-systemd\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544f2794-e894-49e8-b7e9-bb2b030fa972-tmp-dir\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:25.916234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-slash\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-k8s-cni-cncf-io\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-etc-kubernetes\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-systemd\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-config\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysconfig\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-tmp\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysconfig\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-kubernetes\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-var-lib-kubelet\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-socket-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-kubernetes\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-var-lib-kubelet\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-device-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.917045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-socket-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovnkube-config\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cnibin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-sys-fs\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-device-dir\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-os-release\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d60c593-81e3-418c-bbea-0747ab88a80b-sys-fs\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm7x\" (UniqueName: \"kubernetes.io/projected/545d6d14-ef1f-41eb-962a-f4eef7fbade3-kube-api-access-lmm7x\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.916993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-systemd-units\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovn-node-metrics-cert\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-system-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545d6d14-ef1f-41eb-962a-f4eef7fbade3-iptables-alerter-script\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1e5680e-ed52-40a5-8225-973f315728b2-agent-certs\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1e5680e-ed52-40a5-8225-973f315728b2-konnectivity-ca\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79ed7e8f-7393-470b-acca-337a55a5542b-etc-sysctl-d\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.917872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n82r\" (UniqueName: \"kubernetes.io/projected/79ed7e8f-7393-470b-acca-337a55a5542b-kube-api-access-8n82r\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/045408ce-93d2-435e-9f6c-8b9e0e454631-serviceca\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-systemd-units\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a143562-c6dd-444d-b26c-eb6758aa7afd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.917741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1e5680e-ed52-40a5-8225-973f315728b2-konnectivity-ca\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.918667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.918622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-etc-tuned\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.919400 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.919378 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545d6d14-ef1f-41eb-962a-f4eef7fbade3-iptables-alerter-script\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.919400 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.919396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/045408ce-93d2-435e-9f6c-8b9e0e454631-serviceca\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.920746 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.920701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79ed7e8f-7393-470b-acca-337a55a5542b-tmp\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.921406 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.921313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1e5680e-ed52-40a5-8225-973f315728b2-agent-certs\") pod \"konnectivity-agent-thxcm\" (UID: \"e1e5680e-ed52-40a5-8225-973f315728b2\") " pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:25.922013 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.921990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a143562-c6dd-444d-b26c-eb6758aa7afd-ovn-node-metrics-cert\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.924117 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.924095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhqd\" (UniqueName: \"kubernetes.io/projected/5d60c593-81e3-418c-bbea-0747ab88a80b-kube-api-access-vdhqd\") pod \"aws-ebs-csi-driver-node-wt5xj\" (UID: \"5d60c593-81e3-418c-bbea-0747ab88a80b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:25.924577 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.924557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmnkz\" (UniqueName: \"kubernetes.io/projected/3a143562-c6dd-444d-b26c-eb6758aa7afd-kube-api-access-kmnkz\") pod \"ovnkube-node-9sjc7\" (UID: \"3a143562-c6dd-444d-b26c-eb6758aa7afd\") " pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:25.924678 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.924560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pwx\" (UniqueName: \"kubernetes.io/projected/045408ce-93d2-435e-9f6c-8b9e0e454631-kube-api-access-d9pwx\") pod \"node-ca-cfvz2\" (UID: \"045408ce-93d2-435e-9f6c-8b9e0e454631\") " pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:25.924678 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.924612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thl2g\" (UniqueName: \"kubernetes.io/projected/4968ea50-1e0b-49c5-86a5-6c8747e563e9-kube-api-access-thl2g\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:25.925883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.925862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm7x\" (UniqueName: \"kubernetes.io/projected/545d6d14-ef1f-41eb-962a-f4eef7fbade3-kube-api-access-lmm7x\") pod \"iptables-alerter-8sbxs\" (UID: \"545d6d14-ef1f-41eb-962a-f4eef7fbade3\") " pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:25.926558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.926535 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n82r\" (UniqueName: \"kubernetes.io/projected/79ed7e8f-7393-470b-acca-337a55a5542b-kube-api-access-8n82r\") pod \"tuned-sgc2d\" (UID: \"79ed7e8f-7393-470b-acca-337a55a5542b\") " pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:25.976401 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.976345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"1c415e6671b04646ed97d73439cdf78e","Type":"ContainerStarted","Data":"d4be6c78dcee543fa535ff21089cb04cbc66dffa3e118ac0efd910291438d5a0"} Aug 05 17:22:25.977506 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:25.977479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"095232a6940bc44f8134e96f40d4f622","Type":"ContainerStarted","Data":"641d8a24407800f228c7e8a8c92b416739a754c8bbd58218dd1841d67b742159"} Aug 05 17:22:26.018740 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/544f2794-e894-49e8-b7e9-bb2b030fa972-hosts-file\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.018886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/544f2794-e894-49e8-b7e9-bb2b030fa972-hosts-file\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.018886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7c8v\" (UniqueName: \"kubernetes.io/projected/544f2794-e894-49e8-b7e9-bb2b030fa972-kube-api-access-v7c8v\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.018886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-os-release\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.018886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-hostroot\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-hostroot\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-os-release\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-conf-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.018970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-conf-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-daemon-config\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019093 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzrh\" (UniqueName: \"kubernetes.io/projected/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-kube-api-access-2hzrh\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-socket-dir-parent\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-system-cni-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-kubelet-config\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-multus\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-dbus\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-socket-dir-parent\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-system-cni-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-cnibin\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-multus\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-kubelet-config\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdshz\" (UniqueName: \"kubernetes.io/projected/0035653b-7120-4e6d-b28c-b136ba8bd846-kube-api-access-vdshz\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-multus-certs\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544f2794-e894-49e8-b7e9-bb2b030fa972-tmp-dir\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-multus-certs\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-k8s-cni-cncf-io\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.019456 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-etc-kubernetes\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-k8s-cni-cncf-io\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-cnibin\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67a1d7d7-54c6-4f96-8633-3409d07c9a28-dbus\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-etc-kubernetes\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cnibin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.019573 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-os-release\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.019639 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:26.519622256 +0000 UTC m=+3.080907862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544f2794-e894-49e8-b7e9-bb2b030fa972-tmp-dir\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-daemon-config\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-system-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-system-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cni-binary-copy\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-multus-cni-dir\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-netns\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0035653b-7120-4e6d-b28c-b136ba8bd846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-bin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-run-netns\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-kubelet\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-kubelet\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-host-var-lib-cni-bin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-os-release\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.019987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cnibin\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.020057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.020066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.020098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-cni-binary-copy\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.020984 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.020552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0035653b-7120-4e6d-b28c-b136ba8bd846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.027733 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.027703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdshz\" (UniqueName: \"kubernetes.io/projected/0035653b-7120-4e6d-b28c-b136ba8bd846-kube-api-access-vdshz\") pod \"multus-additional-cni-plugins-zjp4n\" (UID: \"0035653b-7120-4e6d-b28c-b136ba8bd846\") " pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.029005 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.028986 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:26.029086 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.029009 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:26.029086 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.029022 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:26.029187 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.029104 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:26.52907394 +0000 UTC m=+3.090359536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:26.031758 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.031731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzrh\" (UniqueName: \"kubernetes.io/projected/b4055cbe-128f-471d-ad0f-0b4f425b8cc9-kube-api-access-2hzrh\") pod \"multus-pvqz7\" (UID: \"b4055cbe-128f-471d-ad0f-0b4f425b8cc9\") " pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.031843 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.031735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7c8v\" (UniqueName: \"kubernetes.io/projected/544f2794-e894-49e8-b7e9-bb2b030fa972-kube-api-access-v7c8v\") pod \"node-resolver-qh92q\" (UID: \"544f2794-e894-49e8-b7e9-bb2b030fa972\") " pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.108242 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.108163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" Aug 05 17:22:26.117158 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.117134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfvz2" Aug 05 17:22:26.125251 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.125230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:26.133056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.133033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sbxs" Aug 05 17:22:26.140638 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.140616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:26.148246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.148227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" Aug 05 17:22:26.154983 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.154966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh92q" Aug 05 17:22:26.162668 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.162652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" Aug 05 17:22:26.168430 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.168413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvqz7" Aug 05 17:22:26.246962 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.246934 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:22:26.422905 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.422819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:26.423048 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.422976 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:26.423048 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.423035 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:27.42302103 +0000 UTC m=+3.984306622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:26.523970 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.523929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:26.524165 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.524082 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:26.524165 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.524136 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:27.524123617 +0000 UTC m=+4.085409209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:26.624518 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.624481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:26.624692 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.624637 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:26.624692 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.624661 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:26.624692 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.624674 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:26.624851 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.624735 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:27.624716103 +0000 UTC m=+4.186001715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:26.848344 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.848292 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:17:25 +0000 UTC" deadline="2028-04-18 07:09:16.67953574 +0000 UTC" Aug 05 17:22:26.848344 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.848341 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14917h46m49.831198772s" Aug 05 17:22:26.849661 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.849620 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4055cbe_128f_471d_ad0f_0b4f425b8cc9.slice/crio-cda34722acaddbbad0316c485725222878973f4da5c6985683d18f03de8574bf WatchSource:0}: Error finding container cda34722acaddbbad0316c485725222878973f4da5c6985683d18f03de8574bf: Status 404 returned error can't find the container with id cda34722acaddbbad0316c485725222878973f4da5c6985683d18f03de8574bf Aug 05 17:22:26.851678 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.851593 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0035653b_7120_4e6d_b28c_b136ba8bd846.slice/crio-9bdc8d41a0cdc7271b7e41fbaf8963299fdbd2765ba0c6144ac6208fc618bff3 WatchSource:0}: Error finding container 9bdc8d41a0cdc7271b7e41fbaf8963299fdbd2765ba0c6144ac6208fc618bff3: Status 404 returned error can't find the container with id 9bdc8d41a0cdc7271b7e41fbaf8963299fdbd2765ba0c6144ac6208fc618bff3 Aug 05 17:22:26.854637 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.854618 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a143562_c6dd_444d_b26c_eb6758aa7afd.slice/crio-f3e137527d2ac0862205440b6275d431655f30db06890fb4f15dbd1832bd4e6b WatchSource:0}: Error finding container f3e137527d2ac0862205440b6275d431655f30db06890fb4f15dbd1832bd4e6b: Status 404 returned error can't find the container with id f3e137527d2ac0862205440b6275d431655f30db06890fb4f15dbd1832bd4e6b Aug 05 17:22:26.855452 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.855424 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045408ce_93d2_435e_9f6c_8b9e0e454631.slice/crio-6fd3a2683c827481b0833003bd69eeaaa86cae77d8ef532e7e2d20f799013900 WatchSource:0}: Error finding container 6fd3a2683c827481b0833003bd69eeaaa86cae77d8ef532e7e2d20f799013900: Status 404 returned error can't find the container with id 6fd3a2683c827481b0833003bd69eeaaa86cae77d8ef532e7e2d20f799013900 Aug 05 17:22:26.857084 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.856949 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d60c593_81e3_418c_bbea_0747ab88a80b.slice/crio-bd44892bf423f1f2740c36e4196c7d4a585a8acdaa9805ea9ff4b8c5bebf6b6b WatchSource:0}: Error finding container bd44892bf423f1f2740c36e4196c7d4a585a8acdaa9805ea9ff4b8c5bebf6b6b: Status 404 returned error can't find the container with id bd44892bf423f1f2740c36e4196c7d4a585a8acdaa9805ea9ff4b8c5bebf6b6b Aug 05 17:22:26.857602 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.857579 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545d6d14_ef1f_41eb_962a_f4eef7fbade3.slice/crio-68f33dc2f1016f5fde617345fe4583865e8ac72ce7b02d7e639a063552ab751a WatchSource:0}: Error finding container 68f33dc2f1016f5fde617345fe4583865e8ac72ce7b02d7e639a063552ab751a: Status 404 returned error can't find the container with id 68f33dc2f1016f5fde617345fe4583865e8ac72ce7b02d7e639a063552ab751a Aug 05 17:22:26.858954 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.858931 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ed7e8f_7393_470b_acca_337a55a5542b.slice/crio-8cd7124288ee7f7edbd8ece4e46baf44219af9bd23d35ab04cdfc0106d89d55b WatchSource:0}: Error finding container 8cd7124288ee7f7edbd8ece4e46baf44219af9bd23d35ab04cdfc0106d89d55b: Status 404 returned error can't find the container with id 8cd7124288ee7f7edbd8ece4e46baf44219af9bd23d35ab04cdfc0106d89d55b Aug 05 17:22:26.859755 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.859731 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e5680e_ed52_40a5_8225_973f315728b2.slice/crio-fea074c170e0acfa8a80c24a96b412bb3844ad7d2c70637a8a58bab859a78787 WatchSource:0}: Error finding container fea074c170e0acfa8a80c24a96b412bb3844ad7d2c70637a8a58bab859a78787: Status 404 returned error can't find the container with id fea074c170e0acfa8a80c24a96b412bb3844ad7d2c70637a8a58bab859a78787 Aug 05 17:22:26.861115 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:26.861090 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544f2794_e894_49e8_b7e9_bb2b030fa972.slice/crio-1cd2a3246a780d67ea6edadc37f858bf2934db65c2792c8e5b6f7a9cd26a3f09 WatchSource:0}: Error finding container 1cd2a3246a780d67ea6edadc37f858bf2934db65c2792c8e5b6f7a9cd26a3f09: Status 404 returned error can't find the container with id 1cd2a3246a780d67ea6edadc37f858bf2934db65c2792c8e5b6f7a9cd26a3f09 Aug 05 17:22:26.972899 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.972877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:26.973061 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:26.973032 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:26.984251 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.984215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvqz7" event={"ID":"b4055cbe-128f-471d-ad0f-0b4f425b8cc9","Type":"ContainerStarted","Data":"cda34722acaddbbad0316c485725222878973f4da5c6985683d18f03de8574bf"} Aug 05 17:22:26.986125 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.986103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh92q" event={"ID":"544f2794-e894-49e8-b7e9-bb2b030fa972","Type":"ContainerStarted","Data":"1cd2a3246a780d67ea6edadc37f858bf2934db65c2792c8e5b6f7a9cd26a3f09"} Aug 05 17:22:26.988661 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.988638 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" event={"ID":"79ed7e8f-7393-470b-acca-337a55a5542b","Type":"ContainerStarted","Data":"8cd7124288ee7f7edbd8ece4e46baf44219af9bd23d35ab04cdfc0106d89d55b"} Aug 05 17:22:26.990216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.990198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sbxs" event={"ID":"545d6d14-ef1f-41eb-962a-f4eef7fbade3","Type":"ContainerStarted","Data":"68f33dc2f1016f5fde617345fe4583865e8ac72ce7b02d7e639a063552ab751a"} Aug 05 17:22:26.993704 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.993683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"f3e137527d2ac0862205440b6275d431655f30db06890fb4f15dbd1832bd4e6b"} Aug 05 17:22:26.995628 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.995607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerStarted","Data":"9bdc8d41a0cdc7271b7e41fbaf8963299fdbd2765ba0c6144ac6208fc618bff3"} Aug 05 17:22:26.998029 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.998009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"1c415e6671b04646ed97d73439cdf78e","Type":"ContainerStarted","Data":"fe8ed4552081de55bb516a7e034899bfbe37d1b565cdf25bec551b97d2035fae"} Aug 05 17:22:26.999407 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:26.999383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfvz2" event={"ID":"045408ce-93d2-435e-9f6c-8b9e0e454631","Type":"ContainerStarted","Data":"6fd3a2683c827481b0833003bd69eeaaa86cae77d8ef532e7e2d20f799013900"} Aug 05 17:22:27.000985 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.000957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thxcm" event={"ID":"e1e5680e-ed52-40a5-8225-973f315728b2","Type":"ContainerStarted","Data":"fea074c170e0acfa8a80c24a96b412bb3844ad7d2c70637a8a58bab859a78787"} Aug 05 17:22:27.001982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.001963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" event={"ID":"5d60c593-81e3-418c-bbea-0747ab88a80b","Type":"ContainerStarted","Data":"bd44892bf423f1f2740c36e4196c7d4a585a8acdaa9805ea9ff4b8c5bebf6b6b"} Aug 05 17:22:27.022354 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.022299 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" podStartSLOduration=2.02228777 podStartE2EDuration="2.02228777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:27.022071309 +0000 UTC m=+3.583356916" watchObservedRunningTime="2026-08-05 17:22:27.02228777 +0000 UTC m=+3.583573404" Aug 05 17:22:27.430828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.430156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:27.430828 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.430333 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:27.430828 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.430399 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:29.430380225 +0000 UTC m=+5.991665820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:27.531255 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.531136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:27.531438 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.531288 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:27.531438 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.531376 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:29.531356329 +0000 UTC m=+6.092641922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:27.632363 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.631745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:27.632363 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.631909 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:27.632363 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.631926 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:27.632363 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.631939 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:27.632363 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.631995 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:29.631976037 +0000 UTC m=+6.193261644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:27.849315 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.849205 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:17:25 +0000 UTC" deadline="2027-12-29 22:06:30.031713281 +0000 UTC" Aug 05 17:22:27.849315 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.849269 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12268h44m2.182449053s" Aug 05 17:22:27.974182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.974151 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:27.974348 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.974293 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:27.974705 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:27.974682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:27.974798 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:27.974780 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:28.021802 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:28.021388 2576 generic.go:356] "Generic (PLEG): container finished" podID="095232a6940bc44f8134e96f40d4f622" containerID="3430be37f3d387752ffec2af4ee93e4655f22003b473038033ad208bffe5834c" exitCode=0 Aug 05 17:22:28.022352 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:28.022308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"095232a6940bc44f8134e96f40d4f622","Type":"ContainerDied","Data":"3430be37f3d387752ffec2af4ee93e4655f22003b473038033ad208bffe5834c"} Aug 05 17:22:28.972515 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:28.972455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:28.972949 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:28.972596 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:29.045133 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.045095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"095232a6940bc44f8134e96f40d4f622","Type":"ContainerStarted","Data":"fec9e0e6b7f76ec69e125ad2fef8471c167962cf3b8d5066fe23957298c227aa"} Aug 05 17:22:29.062855 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.061945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" podStartSLOduration=4.061924566 podStartE2EDuration="4.061924566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:29.061678035 +0000 UTC m=+5.622963653" watchObservedRunningTime="2026-08-05 17:22:29.061924566 +0000 UTC m=+5.623210182" Aug 05 17:22:29.449923 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.449246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:29.449923 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.449429 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:29.449923 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.449496 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:33.44947637 +0000 UTC m=+10.010761965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:29.554400 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.550254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:29.554400 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.552807 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:29.554400 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.552911 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:33.552887942 +0000 UTC m=+10.114173556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:29.651558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.650935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:29.651558 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.651093 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:29.651558 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.651112 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:29.651558 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.651124 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:29.651558 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.651183 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:33.651164721 +0000 UTC m=+10.212450334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:29.975119 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.975045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:29.975591 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.975177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:29.975591 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:29.975219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:29.975591 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:29.975297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:30.972498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:30.972468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:30.972672 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:30.972629 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:31.972064 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:31.972033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:31.972514 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:31.972166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:31.972514 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:31.972226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:31.972514 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:31.972343 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:32.972144 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:32.972109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:32.972663 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:32.972272 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:33.485738 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:33.485128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:33.485738 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.485351 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:33.485738 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.485417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:41.485396796 +0000 UTC m=+18.046682421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:33.586551 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:33.586505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:33.586729 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.586660 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:33.586785 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.586731 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:41.586713139 +0000 UTC m=+18.147998746 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:33.688087 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:33.687477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:33.688087 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.687677 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:33.688087 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.687698 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:33.688087 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.687710 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:33.688087 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.687765 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:41.687746137 +0000 UTC m=+18.249031802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:33.973629 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:33.973549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:33.974116 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.973667 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:33.974116 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:33.974019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:33.974226 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:33.974111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:34.972608 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:34.972521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:34.972824 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:34.972798 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:35.972594 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:35.972558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:35.972965 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:35.972602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:35.972965 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:35.972691 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:35.972965 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:35.972804 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:36.972421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:36.972387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:36.972572 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:36.972518 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:37.972006 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:37.971973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:37.972476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:37.971973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:37.972476 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:37.972111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:37.972476 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:37.972206 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:38.972799 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:38.972764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:38.973218 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:38.972887 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:39.973069 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:39.973034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:39.973559 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:39.973077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:39.973559 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:39.973166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:39.973559 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:39.973292 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:40.971969 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:40.971933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:40.972239 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:40.972078 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:41.550615 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:41.550585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:41.551009 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.550715 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:41.551009 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.550774 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:57.550759636 +0000 UTC m=+34.112045227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:41.651335 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:41.651290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:41.651505 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.651450 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:41.651568 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.651517 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:57.651502262 +0000 UTC m=+34.212787853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:41.751771 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:41.751737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:41.751936 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.751904 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:41.751936 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.751925 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:41.751936 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.751936 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:41.752078 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.751988 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:22:57.751972684 +0000 UTC m=+34.313258295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:41.972100 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:41.972064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:41.972291 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:41.972064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:41.972291 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.972199 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:41.972291 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:41.972241 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:42.971975 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:42.971942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:42.972390 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:42.972070 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:43.973675 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:43.973619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:43.974131 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:43.973734 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:43.974131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:43.973787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:43.974131 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:43.973881 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:44.972089 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:44.972062 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:44.972251 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:44.972167 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:45.973013 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:45.972707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:45.973795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:45.972755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:45.973795 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:45.973113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:45.973795 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:45.973158 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:46.077217 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.077133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh92q" event={"ID":"544f2794-e894-49e8-b7e9-bb2b030fa972","Type":"ContainerStarted","Data":"080dccee21bfcfc9730dd792e78b81b702035db9d0c3a3d74a7d263bf95396cc"} Aug 05 17:22:46.078584 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.078557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" event={"ID":"79ed7e8f-7393-470b-acca-337a55a5542b","Type":"ContainerStarted","Data":"2667fb4f8d0b34740a36d2b466ecc88e866a5a48970aef05474871461863f158"} Aug 05 17:22:46.081439 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"a6186106cfd40e88c50335c7c2fd844118c80fe0e336051915efd345dd83e1d1"} Aug 05 17:22:46.081533 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"48f724f28e2c68bb282d27bfac04dbf5c99d26dc5792b68b0de85bbd3557aa28"} Aug 05 17:22:46.081533 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"04dec3855c1a5103b24ca904c460c42eeb19dafde94029c6889f086d7b044014"} Aug 05 17:22:46.081615 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"382a4ffa660036cf9baa90c3435df76329b57725f9cdd11284f50b63e7abd452"} Aug 05 17:22:46.081615 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"a728fe6a2db529a5a9754a96be2688a83cda047a9e164b5ba9b7bb2f333db4bb"} Aug 05 17:22:46.081687 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.081634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"6f31db79ff9ebcfb4f3ec5e7a4dc32d983fa514e75bd853b84eeb49b300ad6be"} Aug 05 17:22:46.082843 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.082820 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="44007e7d1d192037dca709e9d72d3006dabbeb964eb88c9f67b638c5b9e59380" exitCode=0 Aug 05 17:22:46.082929 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.082891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"44007e7d1d192037dca709e9d72d3006dabbeb964eb88c9f67b638c5b9e59380"} Aug 05 17:22:46.084188 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.084167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfvz2" event={"ID":"045408ce-93d2-435e-9f6c-8b9e0e454631","Type":"ContainerStarted","Data":"2cca379e1a2af61dfdd24a553fe120b980c33cebdad8cb6eba397efe7bd2709c"} Aug 05 17:22:46.085629 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.085609 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thxcm" event={"ID":"e1e5680e-ed52-40a5-8225-973f315728b2","Type":"ContainerStarted","Data":"842012c314899bce4d50034a94b32b75f4363c1b06a92b75173577b5b4afe4d1"} Aug 05 17:22:46.087039 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.087016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" event={"ID":"5d60c593-81e3-418c-bbea-0747ab88a80b","Type":"ContainerStarted","Data":"a05887358e778ac4f0c2ae9142ae3369fc7558b170a5901c2039ba787a59b7dd"} Aug 05 17:22:46.088351 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.088330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvqz7" event={"ID":"b4055cbe-128f-471d-ad0f-0b4f425b8cc9","Type":"ContainerStarted","Data":"a7c3dafe1ca43a1e6cb31dfddf32cc424cf3dc3d18ae013bad3abcb34f1666e3"} Aug 05 17:22:46.096261 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.096227 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qh92q" podStartSLOduration=3.849359428 podStartE2EDuration="22.096215577s" totalImagesPullingTime="18.246856149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:46.095997174 +0000 UTC m=+22.657282789" watchObservedRunningTime="2026-08-05 17:22:46.096215577 +0000 UTC m=+22.657501182" Aug 05 17:22:46.113688 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.113656 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pvqz7" podStartSLOduration=3.783200231 podStartE2EDuration="22.113644489s" totalImagesPullingTime="18.330444258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:46.113557328 +0000 UTC m=+22.674842942" watchObservedRunningTime="2026-08-05 17:22:46.113644489 +0000 UTC m=+22.674930103" Aug 05 17:22:46.147468 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.147420 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-thxcm" podStartSLOduration=7.920034559 podStartE2EDuration="22.147406018s" totalImagesPullingTime="14.227371459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:46.130826977 +0000 UTC m=+22.692112590" watchObservedRunningTime="2026-08-05 17:22:46.147406018 +0000 UTC m=+22.708691642" Aug 05 17:22:46.147588 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.147549 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sgc2d" podStartSLOduration=3.89378721 podStartE2EDuration="22.147543685s" totalImagesPullingTime="18.253756475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:46.147395364 +0000 UTC m=+22.708680979" watchObservedRunningTime="2026-08-05 17:22:46.147543685 +0000 UTC m=+22.708829299" Aug 05 17:22:46.162036 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.162002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cfvz2" podStartSLOduration=4.906907568 podStartE2EDuration="23.161985035s" totalImagesPullingTime="18.255077467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:46.161622188 +0000 UTC m=+22.722907803" watchObservedRunningTime="2026-08-05 17:22:46.161985035 +0000 UTC m=+22.723270650" Aug 05 17:22:46.755780 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.755757 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 17:22:46.878418 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.878263 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:22:46.755775532Z","UUID":"946b6bbd-2e35-406d-9d24-e17c0d558f4a","Handler":null,"Name":"","Endpoint":""} Aug 05 17:22:46.879855 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.879833 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 17:22:46.879980 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.879879 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 17:22:46.972216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:46.972140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:46.972369 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:46.972255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:47.092844 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:47.092805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sbxs" event={"ID":"545d6d14-ef1f-41eb-962a-f4eef7fbade3","Type":"ContainerStarted","Data":"21b3b62c72eb04d82b4dd2d2865185e8dae0ec1bea20745a633fa862a08012e1"} Aug 05 17:22:47.094900 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:47.094867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" event={"ID":"5d60c593-81e3-418c-bbea-0747ab88a80b","Type":"ContainerStarted","Data":"a30bea76a0dd475bc1ef54fddd629fda3d1751308e493d740f6a792acd15af6c"} Aug 05 17:22:47.110052 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:47.110005 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8sbxs" podStartSLOduration=4.856905957 podStartE2EDuration="23.109990655s" totalImagesPullingTime="18.253084698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:47.109671905 +0000 UTC m=+23.670957518" watchObservedRunningTime="2026-08-05 17:22:47.109990655 +0000 UTC m=+23.671276268" Aug 05 17:22:47.973045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:47.973015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:47.973210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:47.973043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:47.973210 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:47.973114 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:47.973210 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:47.973185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:48.101466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.101417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"e65fac52e4e8eaaacfbe5eb92dbea3038960affa839b2603552b21f76d286813"} Aug 05 17:22:48.103992 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.103951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" event={"ID":"5d60c593-81e3-418c-bbea-0747ab88a80b","Type":"ContainerStarted","Data":"b42b8c643a83939635d2fc812fa785760d81d2f8d43e050f0565cb38b48a6157"} Aug 05 17:22:48.128259 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.128212 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt5xj" podStartSLOduration=4.683171304 podStartE2EDuration="25.128188359s" totalImagesPullingTime="20.445017055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:48.127622287 +0000 UTC m=+24.688907901" watchObservedRunningTime="2026-08-05 17:22:48.128188359 +0000 UTC m=+24.689473973" Aug 05 17:22:48.189869 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.189842 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:48.190594 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.190572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:48.972827 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:48.972596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:48.973083 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:48.972939 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:49.106870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:49.106839 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:49.107286 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:49.107270 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-thxcm" Aug 05 17:22:49.972410 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:49.972379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:49.972587 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:49.972380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:49.972587 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:49.972504 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:49.972587 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:49.972566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:50.972111 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:50.972043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:50.972778 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:50.972151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:51.114616 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.114415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" event={"ID":"3a143562-c6dd-444d-b26c-eb6758aa7afd","Type":"ContainerStarted","Data":"002f53b24bdd6597a303b45907527e393c276169877d35d82a45a35799c57470"} Aug 05 17:22:51.114714 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.114688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:51.114781 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.114744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:51.116620 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.116599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerStarted","Data":"6fd4b4e69fa2840ea06e326ef2f37d7b89e2691af8a5c709d33f4b5b1b7de66d"} Aug 05 17:22:51.129531 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.129509 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:51.145479 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.145436 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" podStartSLOduration=8.849194632 podStartE2EDuration="27.145420519s" totalImagesPullingTime="18.296225887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:51.145065742 +0000 UTC m=+27.706351355" watchObservedRunningTime="2026-08-05 17:22:51.145420519 +0000 UTC m=+27.706706133" Aug 05 17:22:51.972290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.972257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:51.972710 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:51.972391 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:51.972710 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:51.972453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:51.972710 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:51.972571 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:52.120994 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.120912 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="6fd4b4e69fa2840ea06e326ef2f37d7b89e2691af8a5c709d33f4b5b1b7de66d" exitCode=0 Aug 05 17:22:52.121165 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.120995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"6fd4b4e69fa2840ea06e326ef2f37d7b89e2691af8a5c709d33f4b5b1b7de66d"} Aug 05 17:22:52.121487 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.121404 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:52.135669 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.135652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:22:52.787532 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.787504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cc44b"] Aug 05 17:22:52.787735 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.787594 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:52.787735 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:52.787690 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:52.789775 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.789748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pf69t"] Aug 05 17:22:52.789907 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.789832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:52.789973 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:52.789904 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:52.790341 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.790307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6x9n7"] Aug 05 17:22:52.790429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:52.790404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:52.790494 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:52.790477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:53.973612 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:53.973582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:53.973964 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:53.973665 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:54.128183 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:54.128154 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="fb3739aa3281ab764d2cdfa7519a8931259eaa1d4c59e9fc37632833b845f860" exitCode=0 Aug 05 17:22:54.128376 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:54.128239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"fb3739aa3281ab764d2cdfa7519a8931259eaa1d4c59e9fc37632833b845f860"} Aug 05 17:22:54.972844 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:54.972672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:54.973008 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:54.972696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:54.973008 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:54.972944 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:54.973008 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:54.972983 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:55.972373 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:55.972342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:55.972807 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:55.972451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cc44b" podUID="67a1d7d7-54c6-4f96-8633-3409d07c9a28" Aug 05 17:22:56.135583 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:56.135551 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="87fe304d11bfe6287903790d4a2c5c37b16668b81201ef31d8d264a8c97dfcf3" exitCode=0 Aug 05 17:22:56.135583 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:56.135588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"87fe304d11bfe6287903790d4a2c5c37b16668b81201ef31d8d264a8c97dfcf3"} Aug 05 17:22:56.973056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:56.973004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:56.973056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:56.973043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:56.973527 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:56.973127 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pf69t" podUID="60f91819-dfa6-4782-9cf9-610f8f3b2088" Aug 05 17:22:56.973527 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:56.973278 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x9n7" podUID="4968ea50-1e0b-49c5-86a5-6c8747e563e9" Aug 05 17:22:57.581827 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.581789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:57.582065 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.581927 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:57.582065 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.582004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:29.581984741 +0000 UTC m=+66.143270336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:22:57.682711 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.682676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:57.682859 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.682845 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:57.682928 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.682916 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret podName:67a1d7d7-54c6-4f96-8633-3409d07c9a28 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:29.68289702 +0000 UTC m=+66.244182626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret") pod "global-pull-secret-syncer-cc44b" (UID: "67a1d7d7-54c6-4f96-8633-3409d07c9a28") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:22:57.759380 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.759354 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeReady" Aug 05 17:22:57.759530 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.759513 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 17:22:57.784031 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.784003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:57.784199 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.784174 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:22:57.784252 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.784203 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:22:57.784252 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.784216 2576 projected.go:196] Error preparing data for projected volume kube-api-access-k8vp7 for pod openshift-network-diagnostics/network-check-target-pf69t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:57.784363 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:57.784268 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7 podName:60f91819-dfa6-4782-9cf9-610f8f3b2088 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:29.784252717 +0000 UTC m=+66.345538329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vp7" (UniqueName: "kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7") pod "network-check-target-pf69t" (UID: "60f91819-dfa6-4782-9cf9-610f8f3b2088") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:22:57.807715 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.807690 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7sh6g"] Aug 05 17:22:57.809727 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.809704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cc456"] Aug 05 17:22:57.809856 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.809839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:57.811759 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.811740 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:57.812447 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.812308 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 17:22:57.812447 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.812357 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 17:22:57.812610 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.812481 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hv8kx\"" Aug 05 17:22:57.814020 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.813992 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 17:22:57.814125 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.814063 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 17:22:57.814125 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.814073 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bkmmp\"" Aug 05 17:22:57.814305 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.814289 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 17:22:57.818396 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.818348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cc456"] Aug 05 17:22:57.826048 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.825990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7sh6g"] Aug 05 17:22:57.972479 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.972445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:22:57.975417 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.975387 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:22:57.985777 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342804d1-3b24-4d69-8016-286343d5ba8b-tmp-dir\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:57.985877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342804d1-3b24-4d69-8016-286343d5ba8b-config-volume\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:57.985877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:57.985877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e033614e-7c60-424e-b2f6-acfa70868a47-cert\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:57.985987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mlf\" (UniqueName: \"kubernetes.io/projected/342804d1-3b24-4d69-8016-286343d5ba8b-kube-api-access-65mlf\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:57.985987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:57.985970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v98d4\" (UniqueName: \"kubernetes.io/projected/e033614e-7c60-424e-b2f6-acfa70868a47-kube-api-access-v98d4\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.087200 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v98d4\" (UniqueName: \"kubernetes.io/projected/e033614e-7c60-424e-b2f6-acfa70868a47-kube-api-access-v98d4\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.087392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342804d1-3b24-4d69-8016-286343d5ba8b-tmp-dir\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.087392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342804d1-3b24-4d69-8016-286343d5ba8b-config-volume\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.087392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.087392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e033614e-7c60-424e-b2f6-acfa70868a47-cert\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.087392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mlf\" (UniqueName: \"kubernetes.io/projected/342804d1-3b24-4d69-8016-286343d5ba8b-kube-api-access-65mlf\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.087659 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:58.087399 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:22:58.087659 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:58.087463 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:22:58.587443008 +0000 UTC m=+35.148728607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:22:58.087659 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/342804d1-3b24-4d69-8016-286343d5ba8b-tmp-dir\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.087953 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.087929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/342804d1-3b24-4d69-8016-286343d5ba8b-config-volume\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.092480 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.092457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e033614e-7c60-424e-b2f6-acfa70868a47-cert\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.096058 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.096031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mlf\" (UniqueName: \"kubernetes.io/projected/342804d1-3b24-4d69-8016-286343d5ba8b-kube-api-access-65mlf\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.096356 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.096335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v98d4\" (UniqueName: \"kubernetes.io/projected/e033614e-7c60-424e-b2f6-acfa70868a47-kube-api-access-v98d4\") pod \"ingress-canary-cc456\" (UID: \"e033614e-7c60-424e-b2f6-acfa70868a47\") " pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.129151 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.129127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cc456" Aug 05 17:22:58.275098 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.275070 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cc456"] Aug 05 17:22:58.279757 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:22:58.279730 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode033614e_7c60_424e_b2f6_acfa70868a47.slice/crio-0a60bb7548a52ef80cfd995a0d41562ffec57d1efe2f38489f04695c716e4e2b WatchSource:0}: Error finding container 0a60bb7548a52ef80cfd995a0d41562ffec57d1efe2f38489f04695c716e4e2b: Status 404 returned error can't find the container with id 0a60bb7548a52ef80cfd995a0d41562ffec57d1efe2f38489f04695c716e4e2b Aug 05 17:22:58.591104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.591012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:58.591264 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:58.591170 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:22:58.591264 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:58.591245 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:22:59.591225193 +0000 UTC m=+36.152510789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:22:58.972363 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.972074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:22:58.972553 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.972130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:22:58.976978 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.976934 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:22:58.977671 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.977202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l8xxn\"" Aug 05 17:22:58.977671 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.977216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 17:22:58.977671 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.977643 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 17:22:58.992464 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:58.992439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8p6dg\"" Aug 05 17:22:59.145148 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:59.145107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cc456" event={"ID":"e033614e-7c60-424e-b2f6-acfa70868a47","Type":"ContainerStarted","Data":"0a60bb7548a52ef80cfd995a0d41562ffec57d1efe2f38489f04695c716e4e2b"} Aug 05 17:22:59.598479 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:22:59.598448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:22:59.598653 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:59.598617 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:22:59.598709 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:22:59.598689 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:23:01.598666372 +0000 UTC m=+38.159951984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:23:01.611408 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:01.611376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:23:01.611796 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:01.611513 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:23:01.611796 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:01.611573 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:23:05.611555779 +0000 UTC m=+42.172841390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:23:02.153943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:02.153907 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="be947f6424ab6d6b5430c7872cd418ffb2ca739b2ac8440be3809f97e6f74332" exitCode=0 Aug 05 17:23:02.154145 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:02.153983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"be947f6424ab6d6b5430c7872cd418ffb2ca739b2ac8440be3809f97e6f74332"} Aug 05 17:23:02.155503 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:02.155478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cc456" event={"ID":"e033614e-7c60-424e-b2f6-acfa70868a47","Type":"ContainerStarted","Data":"242834e8230f494fcc6273745b3a9c88667e9dab539c6bb2ca3f81b7e74c3ef2"} Aug 05 17:23:02.220837 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:02.220787 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cc456" podStartSLOduration=2.328177873 podStartE2EDuration="5.220766905s" totalImagesPullingTime="2.892589032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:02.21993193 +0000 UTC m=+38.781217545" watchObservedRunningTime="2026-08-05 17:23:02.220766905 +0000 UTC m=+38.782052521" Aug 05 17:23:03.162338 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:03.162131 2576 generic.go:356] "Generic (PLEG): container finished" podID="0035653b-7120-4e6d-b28c-b136ba8bd846" containerID="7b0dcdd6f3dfb78114c214e2b987f9fa4938ca48cd2459a7f0519aeaa7606d76" exitCode=0 Aug 05 17:23:03.162728 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:03.162196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerDied","Data":"7b0dcdd6f3dfb78114c214e2b987f9fa4938ca48cd2459a7f0519aeaa7606d76"} Aug 05 17:23:04.167482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:04.167449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" event={"ID":"0035653b-7120-4e6d-b28c-b136ba8bd846","Type":"ContainerStarted","Data":"85d59669dad449a80cecea2f5c099735af8c543362c0a4cc19fdc290355bacf2"} Aug 05 17:23:04.200369 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:04.200307 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjp4n" podStartSLOduration=9.834485223 podStartE2EDuration="40.200290717s" totalImagesPullingTime="30.365805494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:04.199745342 +0000 UTC m=+40.761030956" watchObservedRunningTime="2026-08-05 17:23:04.200290717 +0000 UTC m=+40.761576331" Aug 05 17:23:05.636504 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:05.636473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:23:05.636840 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:05.636569 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:23:05.636840 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:05.636621 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:23:13.636608789 +0000 UTC m=+50.197894381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:23:13.693280 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:13.693234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:23:13.693761 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:13.693369 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:23:13.693761 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:13.693422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:23:29.693409028 +0000 UTC m=+66.254694621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:23:24.148935 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:24.148907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9sjc7" Aug 05 17:23:29.604218 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.604177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:23:29.612660 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.612642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:23:29.614703 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:29.614677 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 05 17:23:29.614765 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:29.614756 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs podName:4968ea50-1e0b-49c5-86a5-6c8747e563e9 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:33.614740319 +0000 UTC m=+130.176025914 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs") pod "network-metrics-daemon-6x9n7" (UID: "4968ea50-1e0b-49c5-86a5-6c8747e563e9") : secret "metrics-daemon-secret" not found Aug 05 17:23:29.705042 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.705006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:23:29.705190 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.705053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:23:29.705236 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:29.705181 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:23:29.705283 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:29.705270 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls podName:342804d1-3b24-4d69-8016-286343d5ba8b nodeName:}" failed. No retries permitted until 2026-08-05 17:24:01.705251808 +0000 UTC m=+98.266537399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls") pod "dns-default-7sh6g" (UID: "342804d1-3b24-4d69-8016-286343d5ba8b") : secret "dns-default-metrics-tls" not found Aug 05 17:23:29.707708 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.707690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:23:29.719795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.719760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67a1d7d7-54c6-4f96-8633-3409d07c9a28-original-pull-secret\") pod \"global-pull-secret-syncer-cc44b\" (UID: \"67a1d7d7-54c6-4f96-8633-3409d07c9a28\") " pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:23:29.781952 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.781921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cc44b" Aug 05 17:23:29.806312 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.806288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:23:29.808937 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.808915 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 17:23:29.818982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.818958 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 17:23:29.830523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.830503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vp7\" (UniqueName: \"kubernetes.io/projected/60f91819-dfa6-4782-9cf9-610f8f3b2088-kube-api-access-k8vp7\") pod \"network-check-target-pf69t\" (UID: \"60f91819-dfa6-4782-9cf9-610f8f3b2088\") " pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:23:29.894459 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.894427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8p6dg\"" Aug 05 17:23:29.902386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.902365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:23:29.908941 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:29.908924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cc44b"] Aug 05 17:23:29.913173 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:29.913153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a1d7d7_54c6_4f96_8633_3409d07c9a28.slice/crio-404743bcc47aa3925a5229a56f6c317dd662cf98f3eef8974c4d688d30e8f455 WatchSource:0}: Error finding container 404743bcc47aa3925a5229a56f6c317dd662cf98f3eef8974c4d688d30e8f455: Status 404 returned error can't find the container with id 404743bcc47aa3925a5229a56f6c317dd662cf98f3eef8974c4d688d30e8f455 Aug 05 17:23:30.026208 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:30.026171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pf69t"] Aug 05 17:23:30.029448 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:30.029420 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f91819_dfa6_4782_9cf9_610f8f3b2088.slice/crio-21967a35c74744cb7c7415046f114d042b1ba161fccee4c4b24e48ff517c88f8 WatchSource:0}: Error finding container 21967a35c74744cb7c7415046f114d042b1ba161fccee4c4b24e48ff517c88f8: Status 404 returned error can't find the container with id 21967a35c74744cb7c7415046f114d042b1ba161fccee4c4b24e48ff517c88f8 Aug 05 17:23:30.225776 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:30.225739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cc44b" event={"ID":"67a1d7d7-54c6-4f96-8633-3409d07c9a28","Type":"ContainerStarted","Data":"404743bcc47aa3925a5229a56f6c317dd662cf98f3eef8974c4d688d30e8f455"} Aug 05 17:23:30.226802 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:30.226776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pf69t" event={"ID":"60f91819-dfa6-4782-9cf9-610f8f3b2088","Type":"ContainerStarted","Data":"21967a35c74744cb7c7415046f114d042b1ba161fccee4c4b24e48ff517c88f8"} Aug 05 17:23:33.828450 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.828415 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2"] Aug 05 17:23:33.830628 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.830607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:33.833774 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.833754 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 17:23:33.833879 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.833754 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d2fj9\"" Aug 05 17:23:33.833939 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.833759 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 17:23:33.834663 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.834643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 05 17:23:33.835720 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.835698 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 05 17:23:33.840123 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.839504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2"] Aug 05 17:23:33.933095 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.933054 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf"] Aug 05 17:23:33.935331 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.935289 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-kz674"] Aug 05 17:23:33.935462 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.935437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:33.937221 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.937203 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8cf7bccb4-x647z"] Aug 05 17:23:33.937364 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.937344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:33.938458 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.938434 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bmndx\"" Aug 05 17:23:33.939163 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.939123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:33.940500 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.940480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:33.940610 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.940518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/487ef48c-f468-461c-b45a-f001a69e0edd-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:33.940610 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.940556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scz7k\" (UniqueName: \"kubernetes.io/projected/487ef48c-f468-461c-b45a-f001a69e0edd-kube-api-access-scz7k\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:33.943052 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 05 17:23:33.943161 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 05 17:23:33.943586 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943561 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 05 17:23:33.943691 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943656 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:23:33.943760 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943693 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 05 17:23:33.944010 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943990 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7smcd\"" Aug 05 17:23:33.944077 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.944020 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 17:23:33.944156 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.943997 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 05 17:23:33.944883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.944526 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 17:23:33.944883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.944530 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 05 17:23:33.944883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.944555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 05 17:23:33.944883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.944766 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 17:23:33.945085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.945040 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 05 17:23:33.945137 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.945090 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 17:23:33.947112 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.946734 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 05 17:23:33.947112 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.946976 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x7sxs\"" Aug 05 17:23:33.948490 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.948444 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf"] Aug 05 17:23:33.956362 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.956315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-kz674"] Aug 05 17:23:33.958140 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.958120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 05 17:23:33.967473 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:33.967425 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8cf7bccb4-x647z"] Aug 05 17:23:34.030205 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.030173 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j"] Aug 05 17:23:34.032287 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.032266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.034542 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.034523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 05 17:23:34.034667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.034628 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 05 17:23:34.034667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.034646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:23:34.034827 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.034754 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 05 17:23:34.035194 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.035174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bnptr\"" Aug 05 17:23:34.038262 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.038242 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xp7c7"] Aug 05 17:23:34.040415 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.040342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.041611 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95b6096f-e41c-4259-9a34-91ac7d5b0521-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.041706 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.041706 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c7e17b-705f-4cca-b600-ce211df20436-serving-cert\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.041706 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-trusted-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.041706 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-stats-auth\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.041911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-tmp\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.041911 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.041758 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:34.041911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.041911 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.041815 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:23:34.541796549 +0000 UTC m=+71.103082156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:34.041911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/487ef48c-f468-461c-b45a-f001a69e0edd-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j"] Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scz7k\" (UniqueName: \"kubernetes.io/projected/487ef48c-f468-461c-b45a-f001a69e0edd-kube-api-access-scz7k\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.041981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kz5j\" (UniqueName: \"kubernetes.io/projected/95b6096f-e41c-4259-9a34-91ac7d5b0521-kube-api-access-7kz5j\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-snapshots\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-service-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs6n\" (UniqueName: \"kubernetes.io/projected/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-kube-api-access-dzs6n\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.042147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-default-certificate\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.042496 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kgh\" (UniqueName: \"kubernetes.io/projected/c5c7e17b-705f-4cca-b600-ce211df20436-kube-api-access-74kgh\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.042496 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.042496 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95b6096f-e41c-4259-9a34-91ac7d5b0521-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.042725 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fqzn2\"" Aug 05 17:23:34.042779 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 05 17:23:34.042832 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.042781 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 05 17:23:34.043141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.043120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 05 17:23:34.043141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.043136 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:23:34.043377 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.043353 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/487ef48c-f468-461c-b45a-f001a69e0edd-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.050529 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.050481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 05 17:23:34.055087 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.055032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xp7c7"] Aug 05 17:23:34.059703 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.059682 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scz7k\" (UniqueName: \"kubernetes.io/projected/487ef48c-f468-461c-b45a-f001a69e0edd-kube-api-access-scz7k\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.143559 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-config\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.143559 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15dd0baa-aeab-4a15-9f72-d595f6b9099d-serving-cert\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.143559 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs6n\" (UniqueName: \"kubernetes.io/projected/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-kube-api-access-dzs6n\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.143818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kz5j\" (UniqueName: \"kubernetes.io/projected/95b6096f-e41c-4259-9a34-91ac7d5b0521-kube-api-access-7kz5j\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.143818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-service-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.143818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kgh\" (UniqueName: \"kubernetes.io/projected/c5c7e17b-705f-4cca-b600-ce211df20436-kube-api-access-74kgh\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.143818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4wt\" (UniqueName: \"kubernetes.io/projected/15dd0baa-aeab-4a15-9f72-d595f6b9099d-kube-api-access-nm4wt\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.143818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-stats-auth\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nm4\" (UniqueName: \"kubernetes.io/projected/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-kube-api-access-46nm4\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95b6096f-e41c-4259-9a34-91ac7d5b0521-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c7e17b-705f-4cca-b600-ce211df20436-serving-cert\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-tmp\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-serving-cert\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.143998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-config\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-snapshots\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-default-certificate\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95b6096f-e41c-4259-9a34-91ac7d5b0521-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-trusted-ca\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-trusted-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-service-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.144361 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-tmp\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.144523 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.144417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:34.64439995 +0000 UTC m=+71.205685547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:34.144987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95b6096f-e41c-4259-9a34-91ac7d5b0521-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.144987 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.144770 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:34.644749371 +0000 UTC m=+71.206034985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:34.144987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.144912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c5c7e17b-705f-4cca-b600-ce211df20436-snapshots\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.145605 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.145582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5c7e17b-705f-4cca-b600-ce211df20436-trusted-ca-bundle\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.146584 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.146561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95b6096f-e41c-4259-9a34-91ac7d5b0521-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.146775 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.146747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c7e17b-705f-4cca-b600-ce211df20436-serving-cert\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.146996 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.146972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-stats-auth\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.147102 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.147087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-default-certificate\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.153001 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.152978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs6n\" (UniqueName: \"kubernetes.io/projected/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-kube-api-access-dzs6n\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.153564 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.153541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kz5j\" (UniqueName: \"kubernetes.io/projected/95b6096f-e41c-4259-9a34-91ac7d5b0521-kube-api-access-7kz5j\") pod \"kube-storage-version-migrator-operator-7878f4b88c-dcfpf\" (UID: \"95b6096f-e41c-4259-9a34-91ac7d5b0521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.153855 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.153668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kgh\" (UniqueName: \"kubernetes.io/projected/c5c7e17b-705f-4cca-b600-ce211df20436-kube-api-access-74kgh\") pod \"insights-operator-77696685b-kz674\" (UID: \"c5c7e17b-705f-4cca-b600-ce211df20436\") " pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.245679 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-config\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.245864 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15dd0baa-aeab-4a15-9f72-d595f6b9099d-serving-cert\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.245864 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4wt\" (UniqueName: \"kubernetes.io/projected/15dd0baa-aeab-4a15-9f72-d595f6b9099d-kube-api-access-nm4wt\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.245864 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nm4\" (UniqueName: \"kubernetes.io/projected/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-kube-api-access-46nm4\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.246027 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-serving-cert\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.246027 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.245965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-config\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.246136 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.246038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-trusted-ca\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.246541 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.246485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-config\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.246690 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.246661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-config\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.246845 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.246826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15dd0baa-aeab-4a15-9f72-d595f6b9099d-trusted-ca\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.248200 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.248175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-serving-cert\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.248350 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.248272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" Aug 05 17:23:34.248350 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.248294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15dd0baa-aeab-4a15-9f72-d595f6b9099d-serving-cert\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.254249 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.254222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nm4\" (UniqueName: \"kubernetes.io/projected/0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb-kube-api-access-46nm4\") pod \"service-ca-operator-784f97bcf6-fd66j\" (UID: \"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.254492 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.254474 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4wt\" (UniqueName: \"kubernetes.io/projected/15dd0baa-aeab-4a15-9f72-d595f6b9099d-kube-api-access-nm4wt\") pod \"console-operator-5ccd6b445f-xp7c7\" (UID: \"15dd0baa-aeab-4a15-9f72-d595f6b9099d\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.261221 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.261201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-kz674" Aug 05 17:23:34.344410 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.344380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" Aug 05 17:23:34.353430 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.353408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:34.548075 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.548038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:34.548271 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.548206 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:34.548353 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.548313 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:23:35.548290829 +0000 UTC m=+72.109576436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:34.649477 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.649444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.649630 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.649498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:34.649630 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.649613 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:34.649701 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.649656 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:35.649631521 +0000 UTC m=+72.210917119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:34.649701 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:34.649688 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:35.649677688 +0000 UTC m=+72.210963286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:34.861696 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:34.861671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-kz674"] Aug 05 17:23:34.865372 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:34.865348 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c7e17b_705f_4cca_b600_ce211df20436.slice/crio-b0a1f53c02c0b0dc5f89c0572008bd11e637fdca97ad86c38943a59084a60c35 WatchSource:0}: Error finding container b0a1f53c02c0b0dc5f89c0572008bd11e637fdca97ad86c38943a59084a60c35: Status 404 returned error can't find the container with id b0a1f53c02c0b0dc5f89c0572008bd11e637fdca97ad86c38943a59084a60c35 Aug 05 17:23:35.079597 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.079529 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j"] Aug 05 17:23:35.082389 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.082303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf"] Aug 05 17:23:35.085019 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.083293 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xp7c7"] Aug 05 17:23:35.085212 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:35.085132 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed6aa2c_eb4b_4268_b371_58ca6a0b8eeb.slice/crio-882224c386c76ab6a3300592ba35c46114006ad1ddba7f903104f34abd922d8e WatchSource:0}: Error finding container 882224c386c76ab6a3300592ba35c46114006ad1ddba7f903104f34abd922d8e: Status 404 returned error can't find the container with id 882224c386c76ab6a3300592ba35c46114006ad1ddba7f903104f34abd922d8e Aug 05 17:23:35.088622 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:35.088596 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15dd0baa_aeab_4a15_9f72_d595f6b9099d.slice/crio-2b001a437a07875742b9d64d9c85306e8c27ea2c9d5154f41b4e63d1624c30fc WatchSource:0}: Error finding container 2b001a437a07875742b9d64d9c85306e8c27ea2c9d5154f41b4e63d1624c30fc: Status 404 returned error can't find the container with id 2b001a437a07875742b9d64d9c85306e8c27ea2c9d5154f41b4e63d1624c30fc Aug 05 17:23:35.089530 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:35.089498 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b6096f_e41c_4259_9a34_91ac7d5b0521.slice/crio-e7e0b41987d655094c126654b75e2f54ac3ea346b57738ea4cd67e927736edf5 WatchSource:0}: Error finding container e7e0b41987d655094c126654b75e2f54ac3ea346b57738ea4cd67e927736edf5: Status 404 returned error can't find the container with id e7e0b41987d655094c126654b75e2f54ac3ea346b57738ea4cd67e927736edf5 Aug 05 17:23:35.240548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.240512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cc44b" event={"ID":"67a1d7d7-54c6-4f96-8633-3409d07c9a28","Type":"ContainerStarted","Data":"9d371b68dc382ff656b161d30a5ab123df557fdd7cfb1e86de62fafd0efb07fc"} Aug 05 17:23:35.241595 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.241572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" event={"ID":"95b6096f-e41c-4259-9a34-91ac7d5b0521","Type":"ContainerStarted","Data":"e7e0b41987d655094c126654b75e2f54ac3ea346b57738ea4cd67e927736edf5"} Aug 05 17:23:35.242654 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.242633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" event={"ID":"15dd0baa-aeab-4a15-9f72-d595f6b9099d","Type":"ContainerStarted","Data":"2b001a437a07875742b9d64d9c85306e8c27ea2c9d5154f41b4e63d1624c30fc"} Aug 05 17:23:35.243578 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.243557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" event={"ID":"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb","Type":"ContainerStarted","Data":"882224c386c76ab6a3300592ba35c46114006ad1ddba7f903104f34abd922d8e"} Aug 05 17:23:35.244961 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.244935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pf69t" event={"ID":"60f91819-dfa6-4782-9cf9-610f8f3b2088","Type":"ContainerStarted","Data":"4444ac673501b807193e77d40822f2941139f0e9072a6d60e7cbd7143eb27612"} Aug 05 17:23:35.245056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.245041 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:23:35.246032 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.246012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-kz674" event={"ID":"c5c7e17b-705f-4cca-b600-ce211df20436","Type":"ContainerStarted","Data":"b0a1f53c02c0b0dc5f89c0572008bd11e637fdca97ad86c38943a59084a60c35"} Aug 05 17:23:35.256361 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.256303 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cc44b" podStartSLOduration=66.462299711 podStartE2EDuration="1m11.256288946s" totalImagesPullingTime="4.793989235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:35.256018099 +0000 UTC m=+71.817303714" watchObservedRunningTime="2026-08-05 17:23:35.256288946 +0000 UTC m=+71.817574561" Aug 05 17:23:35.272901 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.272860 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pf69t" podStartSLOduration=66.612437186 podStartE2EDuration="1m11.272846671s" totalImagesPullingTime="4.660409485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:35.272478627 +0000 UTC m=+71.833764242" watchObservedRunningTime="2026-08-05 17:23:35.272846671 +0000 UTC m=+71.834132284" Aug 05 17:23:35.557338 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.557293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:35.557502 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:35.557457 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:35.557559 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:35.557539 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:23:37.557519006 +0000 UTC m=+74.118804607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:35.658391 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.658045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:35.658391 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:35.658177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:35.658391 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:35.658369 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:37.658348698 +0000 UTC m=+74.219634292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:35.658685 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:35.658421 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:35.658685 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:35.658487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:37.658467264 +0000 UTC m=+74.219752870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:37.574022 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:37.573980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:37.574440 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:37.574136 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:37.574440 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:37.574208 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:23:41.574193052 +0000 UTC m=+78.135478643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:37.675514 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:37.675476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:37.675749 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:37.675567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:37.675749 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:37.675682 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:41.675658488 +0000 UTC m=+78.236944081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:37.675749 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:37.675705 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:37.675749 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:37.675754 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:41.675741589 +0000 UTC m=+78.237027193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:39.261801 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.261759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" event={"ID":"95b6096f-e41c-4259-9a34-91ac7d5b0521","Type":"ContainerStarted","Data":"91459585e758653071e5f976d9581e747612106cd0eb26b47002d8ac83012749"} Aug 05 17:23:39.263541 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.263516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/0.log" Aug 05 17:23:39.263674 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.263562 2576 generic.go:356] "Generic (PLEG): container finished" podID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" containerID="07bb5d270ee60d44845e80743b5b3f6cd17f9175686296d8cfa220c78c0f7997" exitCode=255 Aug 05 17:23:39.263674 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.263646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" event={"ID":"15dd0baa-aeab-4a15-9f72-d595f6b9099d","Type":"ContainerDied","Data":"07bb5d270ee60d44845e80743b5b3f6cd17f9175686296d8cfa220c78c0f7997"} Aug 05 17:23:39.263835 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.263819 2576 scope.go:117] "RemoveContainer" containerID="07bb5d270ee60d44845e80743b5b3f6cd17f9175686296d8cfa220c78c0f7997" Aug 05 17:23:39.265194 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.265173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" event={"ID":"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb","Type":"ContainerStarted","Data":"e3ef21f79081073b21e0c369b6a4b189b778ca97009f09fef656a4c2703bdcc4"} Aug 05 17:23:39.266751 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.266728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-kz674" event={"ID":"c5c7e17b-705f-4cca-b600-ce211df20436","Type":"ContainerStarted","Data":"aeb2d31af9a27c7189544aaa5c7c06396b3b6887d519cc3ee00f2120103b5394"} Aug 05 17:23:39.277776 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.277617 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" podStartSLOduration=2.994618825 podStartE2EDuration="6.277602626s" totalImagesPullingTime="3.282983801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:39.277180565 +0000 UTC m=+75.838466171" watchObservedRunningTime="2026-08-05 17:23:39.277602626 +0000 UTC m=+75.838888240" Aug 05 17:23:39.339815 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.339769 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-kz674" podStartSLOduration=2.843219081 podStartE2EDuration="6.339753211s" totalImagesPullingTime="3.49653413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:39.339091235 +0000 UTC m=+75.900376851" watchObservedRunningTime="2026-08-05 17:23:39.339753211 +0000 UTC m=+75.901038825" Aug 05 17:23:39.355645 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.355586 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" podStartSLOduration=2.073113647 podStartE2EDuration="5.355565321s" totalImagesPullingTime="3.282451674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:39.354226446 +0000 UTC m=+75.915512062" watchObservedRunningTime="2026-08-05 17:23:39.355565321 +0000 UTC m=+75.916850936" Aug 05 17:23:39.521494 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.521426 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jlxzj"] Aug 05 17:23:39.523542 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.523521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.525774 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.525752 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 17:23:39.525885 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.525756 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 17:23:39.526025 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.526010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vf4w2\"" Aug 05 17:23:39.535640 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.535618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jlxzj"] Aug 05 17:23:39.591772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.591737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8caead93-0332-4349-87cc-3cc043567e50-data-volume\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.591911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.591782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8caead93-0332-4349-87cc-3cc043567e50-crio-socket\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.591911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.591889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.591982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.591913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzrt\" (UniqueName: \"kubernetes.io/projected/8caead93-0332-4349-87cc-3cc043567e50-kube-api-access-tlzrt\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.591982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.591964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693026 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.692992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693026 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzrt\" (UniqueName: \"kubernetes.io/projected/8caead93-0332-4349-87cc-3cc043567e50-kube-api-access-tlzrt\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8caead93-0332-4349-87cc-3cc043567e50-data-volume\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693246 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:39.693155 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:39.693246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8caead93-0332-4349-87cc-3cc043567e50-crio-socket\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693246 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:39.693214 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:23:39.693520 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:39.693217 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:40.19319888 +0000 UTC m=+76.754484493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:39.693520 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8caead93-0332-4349-87cc-3cc043567e50-crio-socket\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.693520 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:39.693287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:40.193267195 +0000 UTC m=+76.754552803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : configmap "kube-rbac-proxy" not found Aug 05 17:23:39.693646 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.693528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8caead93-0332-4349-87cc-3cc043567e50-data-volume\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:39.702550 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:39.702529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzrt\" (UniqueName: \"kubernetes.io/projected/8caead93-0332-4349-87cc-3cc043567e50-kube-api-access-tlzrt\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:40.037794 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.037758 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg"] Aug 05 17:23:40.039883 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.039861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" Aug 05 17:23:40.044020 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.042219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 05 17:23:40.044020 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.042499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 05 17:23:40.044020 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.042579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d5kfx\"" Aug 05 17:23:40.048782 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.048759 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg"] Aug 05 17:23:40.097442 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.097408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zjw\" (UniqueName: \"kubernetes.io/projected/cb834346-8e34-4e43-bcde-2e97cb5170cc-kube-api-access-w5zjw\") pod \"migrator-68bc745d87-kmxhg\" (UID: \"cb834346-8e34-4e43-bcde-2e97cb5170cc\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" Aug 05 17:23:40.198077 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.198041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zjw\" (UniqueName: \"kubernetes.io/projected/cb834346-8e34-4e43-bcde-2e97cb5170cc-kube-api-access-w5zjw\") pod \"migrator-68bc745d87-kmxhg\" (UID: \"cb834346-8e34-4e43-bcde-2e97cb5170cc\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" Aug 05 17:23:40.198216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.198090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:40.198216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.198143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:40.198284 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:40.198248 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:23:40.198284 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:40.198277 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:40.198392 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:40.198312 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:41.198293477 +0000 UTC m=+77.759579083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : configmap "kube-rbac-proxy" not found Aug 05 17:23:40.198392 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:40.198346 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:41.198338987 +0000 UTC m=+77.759624579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:40.206967 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.206945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zjw\" (UniqueName: \"kubernetes.io/projected/cb834346-8e34-4e43-bcde-2e97cb5170cc-kube-api-access-w5zjw\") pod \"migrator-68bc745d87-kmxhg\" (UID: \"cb834346-8e34-4e43-bcde-2e97cb5170cc\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" Aug 05 17:23:40.271312 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.271288 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/1.log" Aug 05 17:23:40.271778 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.271763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/0.log" Aug 05 17:23:40.271828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.271796 2576 generic.go:356] "Generic (PLEG): container finished" podID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" exitCode=255 Aug 05 17:23:40.271860 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.271824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" event={"ID":"15dd0baa-aeab-4a15-9f72-d595f6b9099d","Type":"ContainerDied","Data":"bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00"} Aug 05 17:23:40.272834 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.271888 2576 scope.go:117] "RemoveContainer" containerID="07bb5d270ee60d44845e80743b5b3f6cd17f9175686296d8cfa220c78c0f7997" Aug 05 17:23:40.272912 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.272902 2576 scope.go:117] "RemoveContainer" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" Aug 05 17:23:40.273071 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:40.273051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xp7c7_openshift-console-operator(15dd0baa-aeab-4a15-9f72-d595f6b9099d)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podUID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" Aug 05 17:23:40.353185 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.353114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" Aug 05 17:23:40.467695 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:40.467666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg"] Aug 05 17:23:40.470678 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:23:40.470654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb834346_8e34_4e43_bcde_2e97cb5170cc.slice/crio-e02a81b24d6304392a89fbf53e844d1366f7bb4408b532a988c9f8ce00d8e659 WatchSource:0}: Error finding container e02a81b24d6304392a89fbf53e844d1366f7bb4408b532a988c9f8ce00d8e659: Status 404 returned error can't find the container with id e02a81b24d6304392a89fbf53e844d1366f7bb4408b532a988c9f8ce00d8e659 Aug 05 17:23:41.206700 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.206667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:41.206862 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.206720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:41.206862 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.206811 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:23:41.206862 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.206815 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:41.206964 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.206871 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:43.206855639 +0000 UTC m=+79.768141230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:41.206964 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.206886 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:43.206878539 +0000 UTC m=+79.768164130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : configmap "kube-rbac-proxy" not found Aug 05 17:23:41.277476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.277447 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/1.log" Aug 05 17:23:41.277878 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.277837 2576 scope.go:117] "RemoveContainer" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" Aug 05 17:23:41.278055 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.278030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xp7c7_openshift-console-operator(15dd0baa-aeab-4a15-9f72-d595f6b9099d)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podUID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" Aug 05 17:23:41.278858 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.278828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" event={"ID":"cb834346-8e34-4e43-bcde-2e97cb5170cc","Type":"ContainerStarted","Data":"e02a81b24d6304392a89fbf53e844d1366f7bb4408b532a988c9f8ce00d8e659"} Aug 05 17:23:41.475553 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.475478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh92q_544f2794-e894-49e8-b7e9-bb2b030fa972/dns-node-resolver/0.log" Aug 05 17:23:41.609589 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.609552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:41.609751 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.609710 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:41.609806 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.609783 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:23:49.609761703 +0000 UTC m=+86.171047327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:41.710604 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.710571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:41.710780 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:41.710635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:41.710830 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.710773 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:49.710748196 +0000 UTC m=+86.272033802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:41.710876 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.710829 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:41.710922 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:41.710883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:49.710871184 +0000 UTC m=+86.272156776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:42.284265 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:42.284225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" event={"ID":"cb834346-8e34-4e43-bcde-2e97cb5170cc","Type":"ContainerStarted","Data":"a644a3cd8dbbd14c68bd9e1017f0c1eaff2f7fdcfad6e51939df5e8fadc20af5"} Aug 05 17:23:42.284734 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:42.284292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" event={"ID":"cb834346-8e34-4e43-bcde-2e97cb5170cc","Type":"ContainerStarted","Data":"db6015ce606c82360476f7b67c9c67138d1dcd8e1f93ff1a52e6ec28e3b076fc"} Aug 05 17:23:42.303965 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:42.303920 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kmxhg" podStartSLOduration=0.943489388 podStartE2EDuration="2.303905522s" totalImagesPullingTime="1.360416134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:42.302815057 +0000 UTC m=+78.864100670" watchObservedRunningTime="2026-08-05 17:23:42.303905522 +0000 UTC m=+78.865191162" Aug 05 17:23:42.474742 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:42.474713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfvz2_045408ce-93d2-435e-9f6c-8b9e0e454631/node-ca/0.log" Aug 05 17:23:43.074049 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:43.074020 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cc456_e033614e-7c60-424e-b2f6-acfa70868a47/serve-healthcheck-canary/0.log" Aug 05 17:23:43.222154 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:43.222109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:43.222373 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:43.222177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:43.222373 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:43.222246 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:23:43.222373 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:43.222272 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:43.222373 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:43.222348 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:47.222310237 +0000 UTC m=+83.783595833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : configmap "kube-rbac-proxy" not found Aug 05 17:23:43.222373 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:43.222371 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:47.222360262 +0000 UTC m=+83.783645858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:44.354207 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:44.354175 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:44.354605 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:44.354234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:23:44.354605 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:44.354581 2576 scope.go:117] "RemoveContainer" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" Aug 05 17:23:44.354741 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:44.354722 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xp7c7_openshift-console-operator(15dd0baa-aeab-4a15-9f72-d595f6b9099d)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podUID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" Aug 05 17:23:47.253344 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:47.253207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:47.253344 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:47.253263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:47.253758 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:47.253340 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:47.253758 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:47.253390 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:23:47.253758 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:47.253405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:55.253389103 +0000 UTC m=+91.814674714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:47.253758 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:47.253451 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:55.253433425 +0000 UTC m=+91.814719029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : configmap "kube-rbac-proxy" not found Aug 05 17:23:49.674213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:49.674153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:23:49.674657 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:49.674298 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:49.674657 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:49.674378 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls podName:487ef48c-f468-461c-b45a-f001a69e0edd nodeName:}" failed. No retries permitted until 2026-08-05 17:24:05.674360494 +0000 UTC m=+102.235646098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-79qx2" (UID: "487ef48c-f468-461c-b45a-f001a69e0edd") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:23:49.775075 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:49.775044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:49.775210 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:49.775101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:23:49.775210 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:49.775190 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:23:49.775341 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:49.775250 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:05.7752267 +0000 UTC m=+102.336512312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : configmap references non-existent config key: service-ca.crt Aug 05 17:23:49.775341 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:49.775280 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs podName:a61e86fd-ff4d-416b-b21a-6fd2cc238f89 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:05.775268983 +0000 UTC m=+102.336554582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs") pod "router-default-8cf7bccb4-x647z" (UID: "a61e86fd-ff4d-416b-b21a-6fd2cc238f89") : secret "router-metrics-certs-default" not found Aug 05 17:23:55.319762 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:55.319726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:55.320162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:55.319787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:55.320162 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:55.319871 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:23:55.320162 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:55.319948 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls podName:8caead93-0332-4349-87cc-3cc043567e50 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:11.31993301 +0000 UTC m=+107.881218601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jlxzj" (UID: "8caead93-0332-4349-87cc-3cc043567e50") : secret "insights-runtime-extractor-tls" not found Aug 05 17:23:55.320271 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:55.320250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8caead93-0332-4349-87cc-3cc043567e50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:23:55.972557 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:55.972526 2576 scope.go:117] "RemoveContainer" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" Aug 05 17:23:56.329805 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.329728 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:23:56.330153 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.330108 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/1.log" Aug 05 17:23:56.330153 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.330139 2576 generic.go:356] "Generic (PLEG): container finished" podID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" containerID="dac9c6a6b149b02321fd5c81f259c9590a6d32e78363bb8dc2367e5874b25eeb" exitCode=255 Aug 05 17:23:56.330216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.330170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" event={"ID":"15dd0baa-aeab-4a15-9f72-d595f6b9099d","Type":"ContainerDied","Data":"dac9c6a6b149b02321fd5c81f259c9590a6d32e78363bb8dc2367e5874b25eeb"} Aug 05 17:23:56.330254 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.330227 2576 scope.go:117] "RemoveContainer" containerID="bad991251b51029094b74cb5ebf612fc66e7bc0c04a08135f42c1bfca1b02a00" Aug 05 17:23:56.330534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:56.330516 2576 scope.go:117] "RemoveContainer" containerID="dac9c6a6b149b02321fd5c81f259c9590a6d32e78363bb8dc2367e5874b25eeb" Aug 05 17:23:56.330727 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:23:56.330705 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xp7c7_openshift-console-operator(15dd0baa-aeab-4a15-9f72-d595f6b9099d)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podUID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" Aug 05 17:23:57.334244 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:23:57.334219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:24:01.769485 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:01.769433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:24:01.771670 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:01.771650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/342804d1-3b24-4d69-8016-286343d5ba8b-metrics-tls\") pod \"dns-default-7sh6g\" (UID: \"342804d1-3b24-4d69-8016-286343d5ba8b\") " pod="openshift-dns/dns-default-7sh6g" Aug 05 17:24:02.024971 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:02.024894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hv8kx\"" Aug 05 17:24:02.032674 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:02.032660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7sh6g" Aug 05 17:24:02.155177 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:02.155138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7sh6g"] Aug 05 17:24:02.157611 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:02.157584 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342804d1_3b24_4d69_8016_286343d5ba8b.slice/crio-ac063a8e02f2a2979589bcfbc2d7dee872ced8d8b98801880df5732574ef1a1b WatchSource:0}: Error finding container ac063a8e02f2a2979589bcfbc2d7dee872ced8d8b98801880df5732574ef1a1b: Status 404 returned error can't find the container with id ac063a8e02f2a2979589bcfbc2d7dee872ced8d8b98801880df5732574ef1a1b Aug 05 17:24:02.348566 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:02.348535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7sh6g" event={"ID":"342804d1-3b24-4d69-8016-286343d5ba8b","Type":"ContainerStarted","Data":"ac063a8e02f2a2979589bcfbc2d7dee872ced8d8b98801880df5732574ef1a1b"} Aug 05 17:24:04.353673 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.353643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:24:04.354062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.353706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:24:04.354109 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.354092 2576 scope.go:117] "RemoveContainer" containerID="dac9c6a6b149b02321fd5c81f259c9590a6d32e78363bb8dc2367e5874b25eeb" Aug 05 17:24:04.354335 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:04.354292 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xp7c7_openshift-console-operator(15dd0baa-aeab-4a15-9f72-d595f6b9099d)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podUID="15dd0baa-aeab-4a15-9f72-d595f6b9099d" Aug 05 17:24:04.357662 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.357637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7sh6g" event={"ID":"342804d1-3b24-4d69-8016-286343d5ba8b","Type":"ContainerStarted","Data":"13a662ea0663a20e8b21ee9974f7bd925325a6f8bee4de793477e193b7d2bb36"} Aug 05 17:24:04.357800 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.357677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7sh6g" event={"ID":"342804d1-3b24-4d69-8016-286343d5ba8b","Type":"ContainerStarted","Data":"bebb1a7f400cab42040b239a7e075a2979d08e1d3f505656da091390c4ef8c2c"} Aug 05 17:24:04.357800 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.357783 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7sh6g" Aug 05 17:24:04.388008 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:04.387966 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7sh6g" podStartSLOduration=65.809921263 podStartE2EDuration="1m7.387951873s" totalImagesPullingTime="1.57803061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:04.386762151 +0000 UTC m=+100.948047758" watchObservedRunningTime="2026-08-05 17:24:04.387951873 +0000 UTC m=+100.949237487" Aug 05 17:24:05.700628 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.700593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:24:05.702903 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.702876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/487ef48c-f468-461c-b45a-f001a69e0edd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-79qx2\" (UID: \"487ef48c-f468-461c-b45a-f001a69e0edd\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:24:05.801509 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.801472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:05.801642 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.801542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:05.802107 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.802084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-service-ca-bundle\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:05.803704 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.803688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a61e86fd-ff4d-416b-b21a-6fd2cc238f89-metrics-certs\") pod \"router-default-8cf7bccb4-x647z\" (UID: \"a61e86fd-ff4d-416b-b21a-6fd2cc238f89\") " pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:05.944088 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:05.944051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" Aug 05 17:24:06.068849 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.068812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:06.099263 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.099235 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2"] Aug 05 17:24:06.102404 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:06.102379 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487ef48c_f468_461c_b45a_f001a69e0edd.slice/crio-5dcc767333ecf5d90dc966845c827993158bb079429b783d5f7496e3189b431f WatchSource:0}: Error finding container 5dcc767333ecf5d90dc966845c827993158bb079429b783d5f7496e3189b431f: Status 404 returned error can't find the container with id 5dcc767333ecf5d90dc966845c827993158bb079429b783d5f7496e3189b431f Aug 05 17:24:06.219413 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.219376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8cf7bccb4-x647z"] Aug 05 17:24:06.221931 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:06.221907 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61e86fd_ff4d_416b_b21a_6fd2cc238f89.slice/crio-20147a3e38cebad0891cabccadd526cd492152ebd44f604e7f3fc0348633f80e WatchSource:0}: Error finding container 20147a3e38cebad0891cabccadd526cd492152ebd44f604e7f3fc0348633f80e: Status 404 returned error can't find the container with id 20147a3e38cebad0891cabccadd526cd492152ebd44f604e7f3fc0348633f80e Aug 05 17:24:06.253869 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.253823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pf69t" Aug 05 17:24:06.365000 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.364964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" event={"ID":"487ef48c-f468-461c-b45a-f001a69e0edd","Type":"ContainerStarted","Data":"5dcc767333ecf5d90dc966845c827993158bb079429b783d5f7496e3189b431f"} Aug 05 17:24:06.366374 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.366352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8cf7bccb4-x647z" event={"ID":"a61e86fd-ff4d-416b-b21a-6fd2cc238f89","Type":"ContainerStarted","Data":"13b155a6fc04fe61b9ae362e62c90451ae769bc5eb2f4752ab8e0f75425bab8e"} Aug 05 17:24:06.366480 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.366389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8cf7bccb4-x647z" event={"ID":"a61e86fd-ff4d-416b-b21a-6fd2cc238f89","Type":"ContainerStarted","Data":"20147a3e38cebad0891cabccadd526cd492152ebd44f604e7f3fc0348633f80e"} Aug 05 17:24:06.387688 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:06.387643 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8cf7bccb4-x647z" podStartSLOduration=33.387629287 podStartE2EDuration="33.387629287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:06.387507706 +0000 UTC m=+102.948793320" watchObservedRunningTime="2026-08-05 17:24:06.387629287 +0000 UTC m=+102.948914900" Aug 05 17:24:07.070012 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:07.069971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:07.072927 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:07.072905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:07.369620 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:07.369548 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:07.370802 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:07.370784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8cf7bccb4-x647z" Aug 05 17:24:08.373823 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.373798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" event={"ID":"487ef48c-f468-461c-b45a-f001a69e0edd","Type":"ContainerStarted","Data":"085453a736c364cea82916d673459171dc0604f676542f3b6bcff3622ff55b50"} Aug 05 17:24:08.787643 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.787607 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f"] Aug 05 17:24:08.808893 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.808866 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f"] Aug 05 17:24:08.809022 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.808972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:08.811461 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.811436 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nqmnl\"" Aug 05 17:24:08.811577 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.811475 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 05 17:24:08.823615 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.823593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9264f\" (UID: \"95b0f1fc-6504-4228-938b-2dbd60007f66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:08.924823 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:08.924799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9264f\" (UID: \"95b0f1fc-6504-4228-938b-2dbd60007f66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:08.924974 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:08.924925 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Aug 05 17:24:08.924974 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:08.924975 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates podName:95b0f1fc-6504-4228-938b-2dbd60007f66 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:09.424962383 +0000 UTC m=+105.986247976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-9264f" (UID: "95b0f1fc-6504-4228-938b-2dbd60007f66") : secret "prometheus-operator-admission-webhook-tls" not found Aug 05 17:24:09.394273 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:09.394209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-79qx2" podStartSLOduration=34.235782053 podStartE2EDuration="36.394185135s" totalImagesPullingTime="2.158403082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:09.393854676 +0000 UTC m=+105.955140291" watchObservedRunningTime="2026-08-05 17:24:09.394185135 +0000 UTC m=+105.955470750" Aug 05 17:24:09.427471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:09.427440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9264f\" (UID: \"95b0f1fc-6504-4228-938b-2dbd60007f66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:09.429857 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:09.429831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/95b0f1fc-6504-4228-938b-2dbd60007f66-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9264f\" (UID: \"95b0f1fc-6504-4228-938b-2dbd60007f66\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:09.718994 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:09.718963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:09.858928 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:09.858904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f"] Aug 05 17:24:09.861695 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:09.861660 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b0f1fc_6504_4228_938b_2dbd60007f66.slice/crio-28d229d0eadc51bffd4c75c9f809f0da124822ac0546d158f7a35b51e25f15b9 WatchSource:0}: Error finding container 28d229d0eadc51bffd4c75c9f809f0da124822ac0546d158f7a35b51e25f15b9: Status 404 returned error can't find the container with id 28d229d0eadc51bffd4c75c9f809f0da124822ac0546d158f7a35b51e25f15b9 Aug 05 17:24:10.382260 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:10.382222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" event={"ID":"95b0f1fc-6504-4228-938b-2dbd60007f66","Type":"ContainerStarted","Data":"28d229d0eadc51bffd4c75c9f809f0da124822ac0546d158f7a35b51e25f15b9"} Aug 05 17:24:11.340201 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:11.340172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:24:11.342396 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:11.342376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8caead93-0332-4349-87cc-3cc043567e50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jlxzj\" (UID: \"8caead93-0332-4349-87cc-3cc043567e50\") " pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:24:11.633152 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:11.633067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jlxzj" Aug 05 17:24:11.753837 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:11.753809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jlxzj"] Aug 05 17:24:11.756955 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:11.756929 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caead93_0332_4349_87cc_3cc043567e50.slice/crio-0b0231cf58fb3934be8a9660d26c2921b0627b93e171312fce69ec92ce42499a WatchSource:0}: Error finding container 0b0231cf58fb3934be8a9660d26c2921b0627b93e171312fce69ec92ce42499a: Status 404 returned error can't find the container with id 0b0231cf58fb3934be8a9660d26c2921b0627b93e171312fce69ec92ce42499a Aug 05 17:24:12.390911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.390874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlxzj" event={"ID":"8caead93-0332-4349-87cc-3cc043567e50","Type":"ContainerStarted","Data":"85a40098390c6bdca5a91d343b4a96f4eff33136e6011607188629b3021ef398"} Aug 05 17:24:12.391279 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.390928 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlxzj" event={"ID":"8caead93-0332-4349-87cc-3cc043567e50","Type":"ContainerStarted","Data":"0b0231cf58fb3934be8a9660d26c2921b0627b93e171312fce69ec92ce42499a"} Aug 05 17:24:12.392213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.392187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" event={"ID":"95b0f1fc-6504-4228-938b-2dbd60007f66","Type":"ContainerStarted","Data":"ddb0bdad2121d433f6803cce5b9f67b60b1a23ac6bd73146d4d1474d210827db"} Aug 05 17:24:12.392470 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.392446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:12.397055 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.397032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" Aug 05 17:24:12.415540 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:12.415497 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9264f" podStartSLOduration=2.9557355100000002 podStartE2EDuration="4.415485071s" totalImagesPullingTime="1.459749561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:12.41395684 +0000 UTC m=+108.975242456" watchObservedRunningTime="2026-08-05 17:24:12.415485071 +0000 UTC m=+108.976770685" Aug 05 17:24:13.398015 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:13.397979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlxzj" event={"ID":"8caead93-0332-4349-87cc-3cc043567e50","Type":"ContainerStarted","Data":"1c865a60baa5c1735dc1d067b6c32bb96a1d1c2d5aab84dd28299710f53707a4"} Aug 05 17:24:14.362555 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:14.362522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7sh6g" Aug 05 17:24:14.404255 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:14.404209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jlxzj" event={"ID":"8caead93-0332-4349-87cc-3cc043567e50","Type":"ContainerStarted","Data":"f7f31605513597e1d4a83035ce8bde7346586096eec9259387a363f4c53add1b"} Aug 05 17:24:14.430277 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:14.430218 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jlxzj" podStartSLOduration=33.39521829 podStartE2EDuration="35.430197593s" totalImagesPullingTime="2.034979303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:23:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:14.429853246 +0000 UTC m=+110.991138884" watchObservedRunningTime="2026-08-05 17:24:14.430197593 +0000 UTC m=+110.991483212" Aug 05 17:24:16.973055 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:16.973026 2576 scope.go:117] "RemoveContainer" containerID="dac9c6a6b149b02321fd5c81f259c9590a6d32e78363bb8dc2367e5874b25eeb" Aug 05 17:24:17.414964 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:17.414893 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:24:17.415098 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:17.414971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" event={"ID":"15dd0baa-aeab-4a15-9f72-d595f6b9099d","Type":"ContainerStarted","Data":"f626a170cd8a93944e10317d48ef782e807d4f0d3351fd61179c094f520090d6"} Aug 05 17:24:17.415251 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:17.415233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:24:17.437337 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:17.437286 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" podStartSLOduration=40.160840151 podStartE2EDuration="43.437273096s" totalImagesPullingTime="3.276432945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:17.43554097 +0000 UTC m=+113.996826586" watchObservedRunningTime="2026-08-05 17:24:17.437273096 +0000 UTC m=+113.998558745" Aug 05 17:24:17.606360 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:17.606330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xp7c7" Aug 05 17:24:18.358781 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.358749 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr"] Aug 05 17:24:18.360858 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.360842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.372800 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.372781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 05 17:24:18.372911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.372787 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 05 17:24:18.372911 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.372834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nqhnr\"" Aug 05 17:24:18.373754 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.373741 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 17:24:18.389926 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.389887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.390031 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.389949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.390079 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.390024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnlk\" (UniqueName: \"kubernetes.io/projected/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-kube-api-access-9nnlk\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.390079 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.390071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.400626 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.400604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr"] Aug 05 17:24:18.437357 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.437332 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj"] Aug 05 17:24:18.439405 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.439383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.447437 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.447417 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 05 17:24:18.449508 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.449493 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 05 17:24:18.450457 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.450440 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nlj5z\"" Aug 05 17:24:18.450556 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.450455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 05 17:24:18.477973 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.477944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj"] Aug 05 17:24:18.487663 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.487642 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hcrll"] Aug 05 17:24:18.489894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.489880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.490381 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.490466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.490466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnlk\" (UniqueName: \"kubernetes.io/projected/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-kube-api-access-9nnlk\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.490466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.490712 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.490766 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.490947 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.491018 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.490973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.491080 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.491060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxbq\" (UniqueName: \"kubernetes.io/projected/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-api-access-gbxbq\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.491133 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.491100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.491522 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.491502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.493083 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.493178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.493178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dw7jg\"" Aug 05 17:24:18.493377 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493364 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 17:24:18.493422 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493380 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 17:24:18.493600 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.493584 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 17:24:18.507828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.507802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnlk\" (UniqueName: \"kubernetes.io/projected/c7bb9561-ba26-48ae-b21a-3f7a3645c91a-kube-api-access-9nnlk\") pod \"openshift-state-metrics-6bb4d7b4dd-x8qzr\" (UID: \"c7bb9561-ba26-48ae-b21a-3f7a3645c91a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.591978 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.591950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.591978 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.591979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-metrics-client-ca\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.591999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-wtmp\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-sys\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-root\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-tls\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6z2k\" (UniqueName: \"kubernetes.io/projected/269aee79-9d73-44c4-b971-4ee0012c9a95-kube-api-access-j6z2k\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxbq\" (UniqueName: \"kubernetes.io/projected/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-api-access-gbxbq\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-textfile\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:18.592443 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 05 17:24:18.592567 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:18.592549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls podName:0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f nodeName:}" failed. No retries permitted until 2026-08-05 17:24:19.092528899 +0000 UTC m=+115.653814493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-9b9mj" (UID: "0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f") : secret "kube-state-metrics-tls" not found Aug 05 17:24:18.592932 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592932 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.592932 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.592869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.594753 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.594728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.607813 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.607791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxbq\" (UniqueName: \"kubernetes.io/projected/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-api-access-gbxbq\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:18.669254 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.669203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" Aug 05 17:24:18.692974 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.692945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-sys\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.692983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-root\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-root\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-sys\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-tls\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6z2k\" (UniqueName: \"kubernetes.io/projected/269aee79-9d73-44c4-b971-4ee0012c9a95-kube-api-access-j6z2k\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-textfile\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-metrics-client-ca\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-wtmp\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693637 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-wtmp\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693637 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-textfile\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693706 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-accelerators-collector-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.693972 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.693954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269aee79-9d73-44c4-b971-4ee0012c9a95-metrics-client-ca\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.695393 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.695376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.696330 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.696293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269aee79-9d73-44c4-b971-4ee0012c9a95-node-exporter-tls\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.703388 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.703367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6z2k\" (UniqueName: \"kubernetes.io/projected/269aee79-9d73-44c4-b971-4ee0012c9a95-kube-api-access-j6z2k\") pod \"node-exporter-hcrll\" (UID: \"269aee79-9d73-44c4-b971-4ee0012c9a95\") " pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.804425 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.804392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hcrll" Aug 05 17:24:18.806752 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:18.806654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr"] Aug 05 17:24:18.809221 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:18.809198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bb9561_ba26_48ae_b21a_3f7a3645c91a.slice/crio-d397b2928cff89477140f9d5fcb2d26cc0e38ce4e4fdfc5b4adebdfdb33b1db1 WatchSource:0}: Error finding container d397b2928cff89477140f9d5fcb2d26cc0e38ce4e4fdfc5b4adebdfdb33b1db1: Status 404 returned error can't find the container with id d397b2928cff89477140f9d5fcb2d26cc0e38ce4e4fdfc5b4adebdfdb33b1db1 Aug 05 17:24:18.815114 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:18.815000 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269aee79_9d73_44c4_b971_4ee0012c9a95.slice/crio-d9f3b3315e4ad7f8a69ab1ead5a54e74434668ae81d70b1309e67fa0ffbb9bba WatchSource:0}: Error finding container d9f3b3315e4ad7f8a69ab1ead5a54e74434668ae81d70b1309e67fa0ffbb9bba: Status 404 returned error can't find the container with id d9f3b3315e4ad7f8a69ab1ead5a54e74434668ae81d70b1309e67fa0ffbb9bba Aug 05 17:24:19.096982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.096951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:19.099360 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.099314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9b9mj\" (UID: \"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:19.348063 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.347982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" Aug 05 17:24:19.430708 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.430676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcrll" event={"ID":"269aee79-9d73-44c4-b971-4ee0012c9a95","Type":"ContainerStarted","Data":"d9f3b3315e4ad7f8a69ab1ead5a54e74434668ae81d70b1309e67fa0ffbb9bba"} Aug 05 17:24:19.432900 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.432848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" event={"ID":"c7bb9561-ba26-48ae-b21a-3f7a3645c91a","Type":"ContainerStarted","Data":"14f4b6d0ac385d1f15b5f8f7ae89f810f41784abc2227edb2c39b8e61a223d00"} Aug 05 17:24:19.433032 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.432915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" event={"ID":"c7bb9561-ba26-48ae-b21a-3f7a3645c91a","Type":"ContainerStarted","Data":"adf2e5219516fc1e85a7fc04d718fd2532ea5d5bef725ff4d0fbdad01d43410a"} Aug 05 17:24:19.433032 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.432944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" event={"ID":"c7bb9561-ba26-48ae-b21a-3f7a3645c91a","Type":"ContainerStarted","Data":"d397b2928cff89477140f9d5fcb2d26cc0e38ce4e4fdfc5b4adebdfdb33b1db1"} Aug 05 17:24:19.490333 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.490287 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:24:19.493907 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.493885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.500216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.500190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 05 17:24:19.500509 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.500491 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x6vvb\"" Aug 05 17:24:19.501577 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.501556 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 05 17:24:19.501807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.501782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 05 17:24:19.501923 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.501828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 05 17:24:19.502021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.502003 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 05 17:24:19.502073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.502050 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 05 17:24:19.502145 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.502126 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 05 17:24:19.502226 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.502208 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 05 17:24:19.504017 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.503996 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 05 17:24:19.530212 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.530118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:24:19.533801 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.533782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj"] Aug 05 17:24:19.602335 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602335 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602335 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xns\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602571 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602811 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602811 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.602811 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.602635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703486 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703685 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703685 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703685 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703685 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.703894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.704189 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.703939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xns\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.705062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.705031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.705333 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:19.705302 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle podName:f72540ec-d85d-4800-bcab-172e7466eb2b nodeName:}" failed. No retries permitted until 2026-08-05 17:24:20.205281288 +0000 UTC m=+116.766566880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b") : configmap references non-existent config key: ca-bundle.crt Aug 05 17:24:19.705502 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:19.705480 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 05 17:24:19.705573 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:19.705542 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls podName:f72540ec-d85d-4800-bcab-172e7466eb2b nodeName:}" failed. No retries permitted until 2026-08-05 17:24:20.20552506 +0000 UTC m=+116.766810654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b") : secret "alertmanager-main-tls" not found Aug 05 17:24:19.705573 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.705564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.705877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.705855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.707485 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.707416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.707485 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.707461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.707947 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.707919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.708442 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.708419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.708900 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.708865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.709760 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.709736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.709866 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.709738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.710354 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.710283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:19.726876 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:19.726853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xns\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:20.210400 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.210366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:20.210544 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.210459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:20.210544 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:20.210523 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 05 17:24:20.210662 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:24:20.210592 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls podName:f72540ec-d85d-4800-bcab-172e7466eb2b nodeName:}" failed. No retries permitted until 2026-08-05 17:24:21.210576056 +0000 UTC m=+117.771861649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b") : secret "alertmanager-main-tls" not found Aug 05 17:24:20.211315 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.211272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:20.437797 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.437768 2576 generic.go:356] "Generic (PLEG): container finished" podID="269aee79-9d73-44c4-b971-4ee0012c9a95" containerID="f675526ea036c0f029b53c1d7862ceb392d36277ab6b36a7c38ffa9e00e9b03d" exitCode=0 Aug 05 17:24:20.438221 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.437859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcrll" event={"ID":"269aee79-9d73-44c4-b971-4ee0012c9a95","Type":"ContainerDied","Data":"f675526ea036c0f029b53c1d7862ceb392d36277ab6b36a7c38ffa9e00e9b03d"} Aug 05 17:24:20.440841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.440513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" event={"ID":"c7bb9561-ba26-48ae-b21a-3f7a3645c91a","Type":"ContainerStarted","Data":"ff32d0ee32ced7b715d4ce30381e800387ac0ea8bb8d9a120f53449247b99bda"} Aug 05 17:24:20.441815 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.441776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" event={"ID":"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f","Type":"ContainerStarted","Data":"507e061966933030a4589c46a2710cca985971abcbe6d67ec9013f003f536cab"} Aug 05 17:24:20.495838 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:20.495790 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-x8qzr" podStartSLOduration=1.34791739 podStartE2EDuration="2.495773087s" totalImagesPullingTime="1.147855697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:20.494453274 +0000 UTC m=+117.055738889" watchObservedRunningTime="2026-08-05 17:24:20.495773087 +0000 UTC m=+117.057058745" Aug 05 17:24:21.220484 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.220452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:21.222792 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.222771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:21.323205 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.323172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:24:21.448482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.448450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcrll" event={"ID":"269aee79-9d73-44c4-b971-4ee0012c9a95","Type":"ContainerStarted","Data":"7476a60dac6d8ac6d57e464a533d2a615fbc2b5b7e8ce47b3c3fd0ecab6c9394"} Aug 05 17:24:21.448918 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.448510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hcrll" event={"ID":"269aee79-9d73-44c4-b971-4ee0012c9a95","Type":"ContainerStarted","Data":"edbfd46bb0caaafc2c631e7bc78df2deb66c16075564148a6a46e96bdd17a5fd"} Aug 05 17:24:21.450336 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.450295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" event={"ID":"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f","Type":"ContainerStarted","Data":"fd75c8a108b00e19435220c95b103968d6657177dc9b25fde10473bbfb50d625"} Aug 05 17:24:21.450467 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.450363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" event={"ID":"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f","Type":"ContainerStarted","Data":"957bcc22652ea448a0a4299c646d2548bbacf2055b0c2411abbf7346ec10299c"} Aug 05 17:24:21.450467 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.450394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" event={"ID":"0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f","Type":"ContainerStarted","Data":"a894781ce42440ffedfb8546a8cbb008ea2944e407c9fc2eb12731059e256477"} Aug 05 17:24:21.462056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.462038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:24:21.463486 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:21.463464 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72540ec_d85d_4800_bcab_172e7466eb2b.slice/crio-b995ca12fba229ba8fcf43c060f35e62a6d9deaca6ecff17cb519a0ff057374d WatchSource:0}: Error finding container b995ca12fba229ba8fcf43c060f35e62a6d9deaca6ecff17cb519a0ff057374d: Status 404 returned error can't find the container with id b995ca12fba229ba8fcf43c060f35e62a6d9deaca6ecff17cb519a0ff057374d Aug 05 17:24:21.469960 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.469912 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hcrll" podStartSLOduration=2.149984293 podStartE2EDuration="3.469893297s" totalImagesPullingTime="1.2445829s" totalInitContainerRuntime="75.326104ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:21.468601859 +0000 UTC m=+118.029887473" watchObservedRunningTime="2026-08-05 17:24:21.469893297 +0000 UTC m=+118.031178915" Aug 05 17:24:21.493056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:21.492973 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9b9mj" podStartSLOduration=2.20219121 podStartE2EDuration="3.492957455s" totalImagesPullingTime="1.290766245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:21.491030846 +0000 UTC m=+118.052316459" watchObservedRunningTime="2026-08-05 17:24:21.492957455 +0000 UTC m=+118.054243068" Aug 05 17:24:22.456198 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:22.456160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"b995ca12fba229ba8fcf43c060f35e62a6d9deaca6ecff17cb519a0ff057374d"} Aug 05 17:24:23.039723 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.039691 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6494764dfb-b5v66"] Aug 05 17:24:23.041836 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.041820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.046200 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046175 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 05 17:24:23.046361 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 05 17:24:23.046361 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8uc3rgqse7ff3\"" Aug 05 17:24:23.046361 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-92t6c\"" Aug 05 17:24:23.046808 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046790 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 05 17:24:23.046895 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.046859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 05 17:24:23.059104 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.059084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6494764dfb-b5v66"] Aug 05 17:24:23.136958 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.136924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-client-certs\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137119 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.136970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-client-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137119 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.137023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8d35efb3-2d3d-4f49-81de-6666003cf733-audit-log\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137231 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.137134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-metrics-server-audit-profiles\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137231 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.137166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-tls\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137231 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.137196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.137350 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.137252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zch2j\" (UniqueName: \"kubernetes.io/projected/8d35efb3-2d3d-4f49-81de-6666003cf733-kube-api-access-zch2j\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238167 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-metrics-server-audit-profiles\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-tls\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zch2j\" (UniqueName: \"kubernetes.io/projected/8d35efb3-2d3d-4f49-81de-6666003cf733-kube-api-access-zch2j\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-client-certs\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-client-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238549 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8d35efb3-2d3d-4f49-81de-6666003cf733-audit-log\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238712 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8d35efb3-2d3d-4f49-81de-6666003cf733-audit-log\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.238916 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.238895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.239248 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.239223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8d35efb3-2d3d-4f49-81de-6666003cf733-metrics-server-audit-profiles\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.240676 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.240648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-client-certs\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.240787 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.240768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-client-ca-bundle\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.240893 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.240873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8d35efb3-2d3d-4f49-81de-6666003cf733-secret-metrics-server-tls\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.246246 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.246223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zch2j\" (UniqueName: \"kubernetes.io/projected/8d35efb3-2d3d-4f49-81de-6666003cf733-kube-api-access-zch2j\") pod \"metrics-server-6494764dfb-b5v66\" (UID: \"8d35efb3-2d3d-4f49-81de-6666003cf733\") " pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.350934 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.350862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:23.462116 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.462088 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3" exitCode=0 Aug 05 17:24:23.462556 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.462123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3"} Aug 05 17:24:23.473248 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:23.473166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6494764dfb-b5v66"] Aug 05 17:24:23.475976 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:23.475946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d35efb3_2d3d_4f49_81de_6666003cf733.slice/crio-97e13567d7badd5b88a62b87e5e5d2c8385f2d72f448571bd6a5daa1c1900876 WatchSource:0}: Error finding container 97e13567d7badd5b88a62b87e5e5d2c8385f2d72f448571bd6a5daa1c1900876: Status 404 returned error can't find the container with id 97e13567d7badd5b88a62b87e5e5d2c8385f2d72f448571bd6a5daa1c1900876 Aug 05 17:24:24.467269 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.467237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" event={"ID":"8d35efb3-2d3d-4f49-81de-6666003cf733","Type":"ContainerStarted","Data":"97e13567d7badd5b88a62b87e5e5d2c8385f2d72f448571bd6a5daa1c1900876"} Aug 05 17:24:24.641386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.641351 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:24:24.645064 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.645037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.647688 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.647663 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 05 17:24:24.647688 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.647679 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 05 17:24:24.648701 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.648407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 05 17:24:24.648701 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.648421 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649815 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649839 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649907 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 05 17:24:24.650011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.649927 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 05 17:24:24.650461 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.650444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 05 17:24:24.650526 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.650490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8r9tf\"" Aug 05 17:24:24.650582 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.650535 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 05 17:24:24.650582 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.650540 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 05 17:24:24.650803 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.650785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ahs2vu2g29068\"" Aug 05 17:24:24.652535 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.652513 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 05 17:24:24.663904 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.663876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:24:24.752841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.752757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.752973 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.752834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.752973 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.752867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.752973 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.752927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753113 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.752971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753113 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753113 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753213 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753597 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753597 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753597 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.753597 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.753554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkwc\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.854884 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.854850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.854897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkwc\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855498 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.855925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.856212 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.855966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.857429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.856624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.857429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.856635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.857429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.856827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.857429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.857381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.857705 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.857666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.858262 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.858218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.859600 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.859183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.859600 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.859445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.859776 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.859609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.859888 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.859864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.860216 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.860192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.860870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.860445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.860870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.860589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.860870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.860736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.860870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.860753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.861582 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.861543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.861671 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.861597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.870129 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.870108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkwc\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc\") pod \"prometheus-k8s-0\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:24.956545 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:24.956506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:25.349688 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.349658 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:24:25.476504 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.476452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5"} Aug 05 17:24:25.476899 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.476538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8"} Aug 05 17:24:25.476899 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.476585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615"} Aug 05 17:24:25.476899 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.476629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299"} Aug 05 17:24:25.478045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.477996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" event={"ID":"8d35efb3-2d3d-4f49-81de-6666003cf733","Type":"ContainerStarted","Data":"2b13c3490d111b4c99b33a1263a4660a51294acc7f704817b943e44921ca3814"} Aug 05 17:24:25.479454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.479425 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" exitCode=0 Aug 05 17:24:25.479523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.479485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} Aug 05 17:24:25.479574 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.479542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"189401ee9bffe91ac2f4630a09a09e7051a686be331ce6dfd459f77d4c1ea3e9"} Aug 05 17:24:25.495537 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:25.495497 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" podStartSLOduration=0.773332413 podStartE2EDuration="2.49548179s" totalImagesPullingTime="1.722149377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:25.494474727 +0000 UTC m=+122.055760343" watchObservedRunningTime="2026-08-05 17:24:25.49548179 +0000 UTC m=+122.056767404" Aug 05 17:24:26.488296 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:26.488256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19"} Aug 05 17:24:26.488721 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:26.488366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerStarted","Data":"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65"} Aug 05 17:24:26.524369 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:26.524299 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.930262144 podStartE2EDuration="7.524280037s" totalImagesPullingTime="3.576811399s" totalInitContainerRuntime="17.206494ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:26.523236118 +0000 UTC m=+123.084521734" watchObservedRunningTime="2026-08-05 17:24:26.524280037 +0000 UTC m=+123.085565656" Aug 05 17:24:29.504762 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:29.504715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} Aug 05 17:24:29.505186 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:29.504812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} Aug 05 17:24:31.514588 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:31.514553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} Aug 05 17:24:31.514951 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:31.514633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} Aug 05 17:24:31.514951 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:31.514670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} Aug 05 17:24:31.514951 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:31.514709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerStarted","Data":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} Aug 05 17:24:31.564406 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:31.564350 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.621997273 podStartE2EDuration="7.564310258s" totalImagesPullingTime="4.917033899s" totalInitContainerRuntime="25.279086ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:31.562582181 +0000 UTC m=+128.123867797" watchObservedRunningTime="2026-08-05 17:24:31.564310258 +0000 UTC m=+128.125595873" Aug 05 17:24:33.634197 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:33.634160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:24:33.636447 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:33.636418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4968ea50-1e0b-49c5-86a5-6c8747e563e9-metrics-certs\") pod \"network-metrics-daemon-6x9n7\" (UID: \"4968ea50-1e0b-49c5-86a5-6c8747e563e9\") " pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:24:33.789509 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:33.789478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l8xxn\"" Aug 05 17:24:33.796763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:33.796734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x9n7" Aug 05 17:24:33.932009 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:33.931941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6x9n7"] Aug 05 17:24:33.934971 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:24:33.934943 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4968ea50_1e0b_49c5_86a5_6c8747e563e9.slice/crio-7c2e8e43c26a65633a29f284cbdb678e747d07912928ae3caf6eeda7b36ff8f1 WatchSource:0}: Error finding container 7c2e8e43c26a65633a29f284cbdb678e747d07912928ae3caf6eeda7b36ff8f1: Status 404 returned error can't find the container with id 7c2e8e43c26a65633a29f284cbdb678e747d07912928ae3caf6eeda7b36ff8f1 Aug 05 17:24:34.527080 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:34.527042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x9n7" event={"ID":"4968ea50-1e0b-49c5-86a5-6c8747e563e9","Type":"ContainerStarted","Data":"7c2e8e43c26a65633a29f284cbdb678e747d07912928ae3caf6eeda7b36ff8f1"} Aug 05 17:24:34.957147 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:34.957116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:24:35.532790 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:35.532754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x9n7" event={"ID":"4968ea50-1e0b-49c5-86a5-6c8747e563e9","Type":"ContainerStarted","Data":"c3366407f442a3e4f3b52bd124d295abef2d95fac988d3e8b99b651ec28c1275"} Aug 05 17:24:35.532963 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:35.532812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x9n7" event={"ID":"4968ea50-1e0b-49c5-86a5-6c8747e563e9","Type":"ContainerStarted","Data":"68207e6545313fce840636745bbbefd4fec9412487f170c89de6210e5a570266"} Aug 05 17:24:35.550908 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:35.550862 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6x9n7" podStartSLOduration=130.570533279 podStartE2EDuration="2m11.550846542s" totalImagesPullingTime="980.313263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:35.548535056 +0000 UTC m=+132.109820670" watchObservedRunningTime="2026-08-05 17:24:35.550846542 +0000 UTC m=+132.112132156" Aug 05 17:24:43.351906 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:43.351858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:43.352411 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:43.351940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:24:44.571565 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:44.571527 2576 generic.go:356] "Generic (PLEG): container finished" podID="95b6096f-e41c-4259-9a34-91ac7d5b0521" containerID="91459585e758653071e5f976d9581e747612106cd0eb26b47002d8ac83012749" exitCode=0 Aug 05 17:24:44.571930 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:44.571586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" event={"ID":"95b6096f-e41c-4259-9a34-91ac7d5b0521","Type":"ContainerDied","Data":"91459585e758653071e5f976d9581e747612106cd0eb26b47002d8ac83012749"} Aug 05 17:24:44.571930 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:44.571923 2576 scope.go:117] "RemoveContainer" containerID="91459585e758653071e5f976d9581e747612106cd0eb26b47002d8ac83012749" Aug 05 17:24:45.577999 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:45.577958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-dcfpf" event={"ID":"95b6096f-e41c-4259-9a34-91ac7d5b0521","Type":"ContainerStarted","Data":"3c02b7f17bd1f5bab9ee4d635ef99e4cf76d3285db722382a76f20f4ebc9c936"} Aug 05 17:24:49.598043 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.598010 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5c7e17b-705f-4cca-b600-ce211df20436" containerID="aeb2d31af9a27c7189544aaa5c7c06396b3b6887d519cc3ee00f2120103b5394" exitCode=0 Aug 05 17:24:49.598578 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.598082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-kz674" event={"ID":"c5c7e17b-705f-4cca-b600-ce211df20436","Type":"ContainerDied","Data":"aeb2d31af9a27c7189544aaa5c7c06396b3b6887d519cc3ee00f2120103b5394"} Aug 05 17:24:49.598578 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.598430 2576 scope.go:117] "RemoveContainer" containerID="aeb2d31af9a27c7189544aaa5c7c06396b3b6887d519cc3ee00f2120103b5394" Aug 05 17:24:49.599655 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.599635 2576 generic.go:356] "Generic (PLEG): container finished" podID="0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb" containerID="e3ef21f79081073b21e0c369b6a4b189b778ca97009f09fef656a4c2703bdcc4" exitCode=0 Aug 05 17:24:49.599758 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.599726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" event={"ID":"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb","Type":"ContainerDied","Data":"e3ef21f79081073b21e0c369b6a4b189b778ca97009f09fef656a4c2703bdcc4"} Aug 05 17:24:49.600136 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:49.600123 2576 scope.go:117] "RemoveContainer" containerID="e3ef21f79081073b21e0c369b6a4b189b778ca97009f09fef656a4c2703bdcc4" Aug 05 17:24:50.314962 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:50.314929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8cf7bccb4-x647z_a61e86fd-ff4d-416b-b21a-6fd2cc238f89/router/0.log" Aug 05 17:24:50.321624 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:50.321605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cc456_e033614e-7c60-424e-b2f6-acfa70868a47/serve-healthcheck-canary/0.log" Aug 05 17:24:50.605886 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:50.605798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-kz674" event={"ID":"c5c7e17b-705f-4cca-b600-ce211df20436","Type":"ContainerStarted","Data":"2be60e999dfd31881ab3059feddecdf78f7f090e94b56ba6da5dfb7daff3f9e4"} Aug 05 17:24:50.607540 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:24:50.607518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fd66j" event={"ID":"0ed6aa2c-eb4b-4268-b371-58ca6a0b8eeb","Type":"ContainerStarted","Data":"d8055f4300fdd10d956b51b99d9d76412c34d0784287252a276421ae15d2b177"} Aug 05 17:25:03.357629 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:03.357596 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:25:03.362155 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:03.362095 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6494764dfb-b5v66" Aug 05 17:25:24.956938 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:24.956849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:24.972411 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:24.972385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:25.772085 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:25.772056 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:38.840041 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840009 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840399 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="alertmanager" containerID="cri-o://af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" gracePeriod=120 Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840441 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-metric" containerID="cri-o://2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" gracePeriod=120 Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840468 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-web" containerID="cri-o://064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" gracePeriod=120 Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840495 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy" containerID="cri-o://937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" gracePeriod=120 Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840523 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="prom-label-proxy" containerID="cri-o://6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" gracePeriod=120 Aug 05 17:25:38.840648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:38.840511 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="config-reloader" containerID="cri-o://efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" gracePeriod=120 Aug 05 17:25:39.815921 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.815885 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" exitCode=0 Aug 05 17:25:39.815921 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.815917 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" exitCode=0 Aug 05 17:25:39.815921 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.815914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19"} Aug 05 17:25:39.816170 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.815925 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" exitCode=0 Aug 05 17:25:39.816170 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.816009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5"} Aug 05 17:25:39.816170 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.816014 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" exitCode=0 Aug 05 17:25:39.816170 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.816037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615"} Aug 05 17:25:39.816170 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:39.816061 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299"} Aug 05 17:25:40.083031 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.083002 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:25:40.229294 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229294 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229306 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229414 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8xns\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229479 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229676 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.229841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.229704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f72540ec-d85d-4800-bcab-172e7466eb2b\" (UID: \"f72540ec-d85d-4800-bcab-172e7466eb2b\") " Aug 05 17:25:40.230459 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.230144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:40.232275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns" (OuterVolumeSpecName: "kube-api-access-p8xns") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "kube-api-access-p8xns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:25:40.232763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:25:40.232763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232547 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume" (OuterVolumeSpecName: "config-volume") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.232917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out" (OuterVolumeSpecName: "config-out") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:25:40.232917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232858 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:25:40.232917 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232904 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:40.233078 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.232946 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.233253 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.233225 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.234513 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.234484 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.234610 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.234592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.236752 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.236727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.243011 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.242988 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config" (OuterVolumeSpecName: "web-config") pod "f72540ec-d85d-4800-bcab-172e7466eb2b" (UID: "f72540ec-d85d-4800-bcab-172e7466eb2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330478 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-web-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330510 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xns\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-kube-api-access-p8xns\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330525 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330539 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-main-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330550 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-alertmanager-main-db\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330562 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f72540ec-d85d-4800-bcab-172e7466eb2b-tls-assets\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330572 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330574 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330587 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330598 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72540ec-d85d-4800-bcab-172e7466eb2b-metrics-client-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330609 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330621 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-cluster-tls-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330631 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f72540ec-d85d-4800-bcab-172e7466eb2b-config-out\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.330898 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.330641 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f72540ec-d85d-4800-bcab-172e7466eb2b-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:40.822758 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822724 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" exitCode=0 Aug 05 17:25:40.822758 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822750 2576 generic.go:356] "Generic (PLEG): container finished" podID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerID="064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" exitCode=0 Aug 05 17:25:40.822943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822790 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65"} Aug 05 17:25:40.822943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822808 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:25:40.822943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8"} Aug 05 17:25:40.822943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f72540ec-d85d-4800-bcab-172e7466eb2b","Type":"ContainerDied","Data":"b995ca12fba229ba8fcf43c060f35e62a6d9deaca6ecff17cb519a0ff057374d"} Aug 05 17:25:40.822943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.822901 2576 scope.go:117] "RemoveContainer" containerID="6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" Aug 05 17:25:40.830712 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.830692 2576 scope.go:117] "RemoveContainer" containerID="2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" Aug 05 17:25:40.837602 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.837586 2576 scope.go:117] "RemoveContainer" containerID="937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" Aug 05 17:25:40.843976 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.843959 2576 scope.go:117] "RemoveContainer" containerID="064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" Aug 05 17:25:40.850803 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.850778 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:25:40.851492 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.851474 2576 scope.go:117] "RemoveContainer" containerID="efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" Aug 05 17:25:40.857812 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.855133 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:25:40.862067 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.862049 2576 scope.go:117] "RemoveContainer" containerID="af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" Aug 05 17:25:40.868997 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.868978 2576 scope.go:117] "RemoveContainer" containerID="4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3" Aug 05 17:25:40.876043 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876028 2576 scope.go:117] "RemoveContainer" containerID="6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" Aug 05 17:25:40.876303 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.876283 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19\": container with ID starting with 6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19 not found: ID does not exist" containerID="6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" Aug 05 17:25:40.876386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876312 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19"} err="failed to get container status \"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19\": rpc error: code = NotFound desc = could not find container \"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19\": container with ID starting with 6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19 not found: ID does not exist" Aug 05 17:25:40.876386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876363 2576 scope.go:117] "RemoveContainer" containerID="2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" Aug 05 17:25:40.876586 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.876569 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65\": container with ID starting with 2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65 not found: ID does not exist" containerID="2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" Aug 05 17:25:40.876622 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876591 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65"} err="failed to get container status \"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65\": rpc error: code = NotFound desc = could not find container \"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65\": container with ID starting with 2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65 not found: ID does not exist" Aug 05 17:25:40.876622 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876608 2576 scope.go:117] "RemoveContainer" containerID="937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" Aug 05 17:25:40.876856 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.876838 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5\": container with ID starting with 937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5 not found: ID does not exist" containerID="937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" Aug 05 17:25:40.876908 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876859 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5"} err="failed to get container status \"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5\": rpc error: code = NotFound desc = could not find container \"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5\": container with ID starting with 937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5 not found: ID does not exist" Aug 05 17:25:40.876908 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.876874 2576 scope.go:117] "RemoveContainer" containerID="064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" Aug 05 17:25:40.877098 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.877080 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8\": container with ID starting with 064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8 not found: ID does not exist" containerID="064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" Aug 05 17:25:40.877136 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877103 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8"} err="failed to get container status \"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8\": rpc error: code = NotFound desc = could not find container \"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8\": container with ID starting with 064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8 not found: ID does not exist" Aug 05 17:25:40.877136 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877119 2576 scope.go:117] "RemoveContainer" containerID="efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" Aug 05 17:25:40.877385 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.877339 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615\": container with ID starting with efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615 not found: ID does not exist" containerID="efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" Aug 05 17:25:40.877385 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877358 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615"} err="failed to get container status \"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615\": rpc error: code = NotFound desc = could not find container \"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615\": container with ID starting with efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615 not found: ID does not exist" Aug 05 17:25:40.877385 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877371 2576 scope.go:117] "RemoveContainer" containerID="af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" Aug 05 17:25:40.877586 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.877567 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299\": container with ID starting with af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299 not found: ID does not exist" containerID="af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" Aug 05 17:25:40.877625 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877590 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299"} err="failed to get container status \"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299\": rpc error: code = NotFound desc = could not find container \"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299\": container with ID starting with af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299 not found: ID does not exist" Aug 05 17:25:40.877625 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877603 2576 scope.go:117] "RemoveContainer" containerID="4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3" Aug 05 17:25:40.877812 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:40.877797 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3\": container with ID starting with 4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3 not found: ID does not exist" containerID="4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3" Aug 05 17:25:40.877849 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877815 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3"} err="failed to get container status \"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3\": rpc error: code = NotFound desc = could not find container \"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3\": container with ID starting with 4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3 not found: ID does not exist" Aug 05 17:25:40.877849 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.877827 2576 scope.go:117] "RemoveContainer" containerID="6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19" Aug 05 17:25:40.878039 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878023 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19"} err="failed to get container status \"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19\": rpc error: code = NotFound desc = could not find container \"6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19\": container with ID starting with 6e3e4df8d49aa4cc93d8e1e5051cbf543a5d9649515601b03c309ae29dfd5a19 not found: ID does not exist" Aug 05 17:25:40.878083 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878039 2576 scope.go:117] "RemoveContainer" containerID="2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65" Aug 05 17:25:40.878231 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878216 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65"} err="failed to get container status \"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65\": rpc error: code = NotFound desc = could not find container \"2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65\": container with ID starting with 2d1294b5b741d4b0e5e0796e51b1a8683a026e210740c189e482fe26e9a1eb65 not found: ID does not exist" Aug 05 17:25:40.878276 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878231 2576 scope.go:117] "RemoveContainer" containerID="937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5" Aug 05 17:25:40.878457 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878437 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5"} err="failed to get container status \"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5\": rpc error: code = NotFound desc = could not find container \"937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5\": container with ID starting with 937ceed6684048beed9080818b2562313eebf517c7962a259d86bd2b1f2c0ef5 not found: ID does not exist" Aug 05 17:25:40.878504 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878457 2576 scope.go:117] "RemoveContainer" containerID="064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8" Aug 05 17:25:40.878667 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878651 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8"} err="failed to get container status \"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8\": rpc error: code = NotFound desc = could not find container \"064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8\": container with ID starting with 064e7bbcedc7bf39080c4b84410204a16a5cc8dbe26304913b1548a074aa4ff8 not found: ID does not exist" Aug 05 17:25:40.878717 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878667 2576 scope.go:117] "RemoveContainer" containerID="efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615" Aug 05 17:25:40.878890 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878875 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615"} err="failed to get container status \"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615\": rpc error: code = NotFound desc = could not find container \"efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615\": container with ID starting with efa7435117cf724bfcf28f03213b6cbe5f0f6147b7933a0c8b941066c9425615 not found: ID does not exist" Aug 05 17:25:40.878927 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.878891 2576 scope.go:117] "RemoveContainer" containerID="af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299" Aug 05 17:25:40.879105 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.879083 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299"} err="failed to get container status \"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299\": rpc error: code = NotFound desc = could not find container \"af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299\": container with ID starting with af4961360ac6f5bba1c077aac1f833cfc75f7cbda6415e7e20974206bb562299 not found: ID does not exist" Aug 05 17:25:40.879105 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.879104 2576 scope.go:117] "RemoveContainer" containerID="4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3" Aug 05 17:25:40.879349 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:40.879333 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3"} err="failed to get container status \"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3\": rpc error: code = NotFound desc = could not find container \"4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3\": container with ID starting with 4fb5107ac62f842907de4137a3b55211d4af7c82118fa2a61f5cff832820d5f3 not found: ID does not exist" Aug 05 17:25:41.977042 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:41.977009 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" path="/var/lib/kubelet/pods/f72540ec-d85d-4800-bcab-172e7466eb2b/volumes" Aug 05 17:25:43.142648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.142613 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:43.143043 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.142972 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="prometheus" containerID="cri-o://be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" gracePeriod=600 Aug 05 17:25:43.143043 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.143012 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" gracePeriod=600 Aug 05 17:25:43.143043 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.143032 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="config-reloader" containerID="cri-o://4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" gracePeriod=600 Aug 05 17:25:43.143218 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.143045 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="thanos-sidecar" containerID="cri-o://27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" gracePeriod=600 Aug 05 17:25:43.143218 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.143005 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy" containerID="cri-o://d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" gracePeriod=600 Aug 05 17:25:43.143218 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.143046 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-web" containerID="cri-o://e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" gracePeriod=600 Aug 05 17:25:43.383677 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.383654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.454266 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454293 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454361 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454425 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454449 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454477 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlkwc\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454613 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454643 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454707 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454719 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.454763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454763 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454819 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454852 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls\") pod \"79e756c4-9867-43d8-ab89-58bafcba109e\" (UID: \"79e756c4-9867-43d8-ab89-58bafcba109e\") " Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.454942 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.455158 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.455299 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.455181 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.455673 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.455587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.456968 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.456937 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.456968 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.456950 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.457446 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.457199 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out" (OuterVolumeSpecName: "config-out") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:25:43.457446 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.457223 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.458079 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.457815 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.458079 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.458017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:25:43.458354 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.458187 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:25:43.458720 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.458480 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.458720 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.458640 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.459107 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.459055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.459292 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.459260 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config" (OuterVolumeSpecName: "config") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.459391 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.459352 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:25:43.459505 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.459484 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.459738 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.459719 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc" (OuterVolumeSpecName: "kube-api-access-dlkwc") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "kube-api-access-dlkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:25:43.460436 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.460412 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.460753 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.460729 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.468108 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.468086 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config" (OuterVolumeSpecName: "web-config") pod "79e756c4-9867-43d8-ab89-58bafcba109e" (UID: "79e756c4-9867-43d8-ab89-58bafcba109e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:25:43.555550 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555527 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-metrics-client-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555550 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555549 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-db\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555559 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555568 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555578 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-kube-rbac-proxy\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555586 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555596 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555606 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555616 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-tls-assets\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555624 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-metrics-client-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555633 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555642 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555650 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-secret-grpc-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555660 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkwc\" (UniqueName: \"kubernetes.io/projected/79e756c4-9867-43d8-ab89-58bafcba109e-kube-api-access-dlkwc\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555667 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79e756c4-9867-43d8-ab89-58bafcba109e-web-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555675 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79e756c4-9867-43d8-ab89-58bafcba109e-config-out\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.555680 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555682 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.556127 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.555690 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79e756c4-9867-43d8-ab89-58bafcba109e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838442 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838467 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838474 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838480 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838485 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838490 2576 generic.go:356] "Generic (PLEG): container finished" podID="79e756c4-9867-43d8-ab89-58bafcba109e" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" exitCode=0 Aug 05 17:25:43.838536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838540 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838669 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} Aug 05 17:25:43.838982 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.838801 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79e756c4-9867-43d8-ab89-58bafcba109e","Type":"ContainerDied","Data":"189401ee9bffe91ac2f4630a09a09e7051a686be331ce6dfd459f77d4c1ea3e9"} Aug 05 17:25:43.846887 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.846870 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.854482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.854466 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.861519 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.861501 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.864623 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.864599 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:43.870185 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.870163 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.870764 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.870743 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:43.878063 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.878040 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.885640 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.885616 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.892981 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.892964 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.893257 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.893237 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.893305 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893266 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.893305 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893285 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.893538 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.893521 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.893603 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893547 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.893603 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893568 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.893820 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.893803 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.893876 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893828 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.893876 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.893851 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.894120 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.894103 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.894154 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894125 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.894154 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894142 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.894419 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.894402 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.894471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894423 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.894471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894435 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.894678 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.894665 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.894735 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.894735 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894692 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.894948 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:25:43.894933 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.894991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894951 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.894991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.894965 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.895211 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895188 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.895259 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895214 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.895421 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895403 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.895471 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895422 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.895644 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895624 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.895713 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895647 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.895904 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895887 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.895948 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.895904 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.896098 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896082 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.896137 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896099 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.896303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896285 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.896379 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896305 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.896548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896533 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.896548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896548 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.896781 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896763 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.896781 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896780 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.897015 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.896998 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.897075 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897016 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.897218 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897195 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.897260 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897219 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.897562 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897544 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.897562 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897561 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.897769 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897755 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.897815 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897770 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.897944 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897923 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.897981 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.897945 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.898182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898164 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.898182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898181 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.898415 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898391 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.898464 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898416 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.898639 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898616 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.898691 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898639 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.898845 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898828 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.898894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.898845 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.899048 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899027 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.899048 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899045 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.899301 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899279 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.899301 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899302 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.899545 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899525 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.899632 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899545 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.899775 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899745 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.899775 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.899768 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.900350 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.900238 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.900350 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.900279 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.900763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.900732 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.900861 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.900762 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.901249 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901229 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.901249 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901257 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.901523 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901505 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.901584 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901524 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.901745 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901722 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.901745 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901739 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.901951 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901934 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.902002 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.901953 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.902040 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902024 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:43.902163 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902133 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.902163 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902157 2576 scope.go:117] "RemoveContainer" containerID="a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe" Aug 05 17:25:43.902352 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902337 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="init-config-reloader" Aug 05 17:25:43.902352 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902342 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe"} err="failed to get container status \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": rpc error: code = NotFound desc = could not find container \"a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe\": container with ID starting with a8841697ccb46a427c33175cbd6611be162179ae8ea1f5bb097f495d0544a9fe not found: ID does not exist" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902356 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="init-config-reloader" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902360 2576 scope.go:117] "RemoveContainer" containerID="d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902373 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-metric" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902379 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-metric" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902387 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="config-reloader" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="config-reloader" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902402 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902408 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902418 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="config-reloader" Aug 05 17:25:43.902434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902425 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="config-reloader" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902437 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="prom-label-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902446 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="prom-label-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902455 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="thanos-sidecar" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902463 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="thanos-sidecar" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902481 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="init-config-reloader" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902488 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="init-config-reloader" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902498 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902507 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902521 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902529 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902537 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="prometheus" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902542 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="prometheus" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902549 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-thanos" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902554 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-thanos" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902560 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="alertmanager" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902565 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="alertmanager" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902570 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902575 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902583 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a"} err="failed to get container status \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": rpc error: code = NotFound desc = could not find container \"d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a\": container with ID starting with d8277b05fcb1877c482574294c3494be069444f3be741f7bb53e76f0d65c066a not found: ID does not exist" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902598 2576 scope.go:117] "RemoveContainer" containerID="e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902634 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902643 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy-metric" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902651 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902657 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902663 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="config-reloader" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902669 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="prometheus" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902675 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-thanos" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902679 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="kube-rbac-proxy-web" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902688 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="thanos-sidecar" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902697 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="prom-label-proxy" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902705 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72540ec-d85d-4800-bcab-172e7466eb2b" containerName="alertmanager" Aug 05 17:25:43.902726 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902713 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" containerName="config-reloader" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902792 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf"} err="failed to get container status \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": rpc error: code = NotFound desc = could not find container \"e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf\": container with ID starting with e6d1bbb6195a8578ea487d06fed325e6eb882b0cacce6a7465d25a6609c190cf not found: ID does not exist" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.902808 2576 scope.go:117] "RemoveContainer" containerID="27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903029 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613"} err="failed to get container status \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": rpc error: code = NotFound desc = could not find container \"27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613\": container with ID starting with 27e53ab59dac315c349e4485455247b00a9be62db76a872dcee3985a82fc3613 not found: ID does not exist" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903044 2576 scope.go:117] "RemoveContainer" containerID="4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903256 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303"} err="failed to get container status \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": rpc error: code = NotFound desc = could not find container \"4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303\": container with ID starting with 4825357dd41f81e1223063c1df9a10db245a5f46e2b7502b27b81e0a11ce8303 not found: ID does not exist" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903272 2576 scope.go:117] "RemoveContainer" containerID="be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903493 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff"} err="failed to get container status \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": rpc error: code = NotFound desc = could not find container \"be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff\": container with ID starting with be8103e8a0067f39bb6737bff378ab17e3658a4ccab924e5561b85b76c3d6bff not found: ID does not exist" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903508 2576 scope.go:117] "RemoveContainer" containerID="f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899" Aug 05 17:25:43.903826 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.903725 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899"} err="failed to get container status \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": rpc error: code = NotFound desc = could not find container \"f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899\": container with ID starting with f1614b862feb66bed2cdb64f6c74e170f7d9eb5d4b608794d9d93d450d863899 not found: ID does not exist" Aug 05 17:25:43.906757 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.906739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.909472 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.909453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 05 17:25:43.909558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.909536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 05 17:25:43.909660 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.909642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ahs2vu2g29068\"" Aug 05 17:25:43.911105 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.911087 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 05 17:25:43.913057 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913038 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 05 17:25:43.913057 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913053 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 05 17:25:43.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 05 17:25:43.913534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 05 17:25:43.913723 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8r9tf\"" Aug 05 17:25:43.913823 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 05 17:25:43.913823 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.913742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 05 17:25:43.914053 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.914035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 05 17:25:43.914175 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.914072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 05 17:25:43.914239 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.914038 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 05 17:25:43.915303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.915281 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 05 17:25:43.920168 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.920149 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 05 17:25:43.926073 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.926046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:43.959143 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959268 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-web-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959268 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959268 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959268 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959491 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-config-out\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv8cd\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-kube-api-access-jv8cd\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.959772 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.960005 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.959779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:43.978386 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:43.978352 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e756c4-9867-43d8-ab89-58bafcba109e" path="/var/lib/kubelet/pods/79e756c4-9867-43d8-ab89-58bafcba109e/volumes" Aug 05 17:25:44.060511 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060659 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060659 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-web-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060795 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.060987 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.060970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-config-out\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061290 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv8cd\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-kube-api-access-jv8cd\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061717 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061717 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.061717 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.063786 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.061991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.064111 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.064807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225af48d-5824-48a0-931d-6259cf4ac5da-config-out\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.064807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.064807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.064807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.064861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065096 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.065039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065096 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.065042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065480 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.065441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-web-config\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065579 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.065544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.065761 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.065739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.066026 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.066005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.066138 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.066122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225af48d-5824-48a0-931d-6259cf4ac5da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.066846 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.066823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.066894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.066866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225af48d-5824-48a0-931d-6259cf4ac5da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.068596 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.068580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv8cd\" (UniqueName: \"kubernetes.io/projected/225af48d-5824-48a0-931d-6259cf4ac5da-kube-api-access-jv8cd\") pod \"prometheus-k8s-0\" (UID: \"225af48d-5824-48a0-931d-6259cf4ac5da\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.218932 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.218885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:25:44.379352 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.379314 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:25:44.381662 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:25:44.381636 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225af48d_5824_48a0_931d_6259cf4ac5da.slice/crio-8561d3baacab2113fdf9ddd626fbe3f32278ff933fe48aa693d2bda9cbde90ff WatchSource:0}: Error finding container 8561d3baacab2113fdf9ddd626fbe3f32278ff933fe48aa693d2bda9cbde90ff: Status 404 returned error can't find the container with id 8561d3baacab2113fdf9ddd626fbe3f32278ff933fe48aa693d2bda9cbde90ff Aug 05 17:25:44.843827 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.843794 2576 generic.go:356] "Generic (PLEG): container finished" podID="225af48d-5824-48a0-931d-6259cf4ac5da" containerID="40ae29043b98a02c536dfa9348f0baf652574a4d87d398d702375d8711cf4b21" exitCode=0 Aug 05 17:25:44.844029 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.843885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerDied","Data":"40ae29043b98a02c536dfa9348f0baf652574a4d87d398d702375d8711cf4b21"} Aug 05 17:25:44.844029 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:44.843965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"8561d3baacab2113fdf9ddd626fbe3f32278ff933fe48aa693d2bda9cbde90ff"} Aug 05 17:25:45.853008 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.852977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"d43ef814ca504a41736a84327f9873ccf26b977725af7a863a2a64bacab949e0"} Aug 05 17:25:45.853394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.853038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"24a05636902726efc13c3a24d25babce497ebbe9adb5f7af59890462c2034609"} Aug 05 17:25:45.853394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.853072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"72e1001223f19f96f1f89ff7ecd5c5608e81ed691e9094827d5147ab84d5ae55"} Aug 05 17:25:45.853394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.853098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"0f6228bdf959253df312d772ebd5826257157accecbe3588cb0a0454536ddbd6"} Aug 05 17:25:45.853394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.853121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"33c11a9be8561f848712677531c6effbd95553ac71a97a21dd9993528df67585"} Aug 05 17:25:45.853394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.853144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225af48d-5824-48a0-931d-6259cf4ac5da","Type":"ContainerStarted","Data":"615e0e13a6caba1c2f5fa958ef1d59ac691a629848c5546f7f9931b7a3d8f561"} Aug 05 17:25:45.890190 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:45.890134 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.870307663 podStartE2EDuration="2.890109103s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.80144ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:45.887478758 +0000 UTC m=+202.448764372" watchObservedRunningTime="2026-08-05 17:25:45.890109103 +0000 UTC m=+202.451394717" Aug 05 17:25:49.219701 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:25:49.219669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:26:44.219281 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:26:44.219230 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:26:44.235139 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:26:44.235112 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:26:45.099200 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:26:45.099170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:27:23.812983 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:27:23.812953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:27:23.814644 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:27:23.814616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:27:23.824555 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:27:23.824535 2576 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 17:28:23.352757 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.352677 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bcf6t"] Aug 05 17:28:23.356372 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.356351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.358648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.358619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 05 17:28:23.358648 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.358630 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 05 17:28:23.364169 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.364146 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bcf6t"] Aug 05 17:28:23.377519 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.377490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.377623 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.377605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq2r\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-kube-api-access-kgq2r\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.478062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.478034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.478195 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.478093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq2r\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-kube-api-access-kgq2r\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.486340 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.486291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.486708 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.486691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq2r\" (UniqueName: \"kubernetes.io/projected/d0054a8e-de1d-4ae1-989f-a272169a9d9f-kube-api-access-kgq2r\") pod \"cert-manager-55b4d954d5-bcf6t\" (UID: \"d0054a8e-de1d-4ae1-989f-a272169a9d9f\") " pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.674738 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.674659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bcf6t" Aug 05 17:28:23.797841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.797817 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bcf6t"] Aug 05 17:28:23.800599 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:28:23.800573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0054a8e_de1d_4ae1_989f_a272169a9d9f.slice/crio-60c1866a264eb11d65fba04a2afe36e8890330c3db67a0b20c054be70cf685fd WatchSource:0}: Error finding container 60c1866a264eb11d65fba04a2afe36e8890330c3db67a0b20c054be70cf685fd: Status 404 returned error can't find the container with id 60c1866a264eb11d65fba04a2afe36e8890330c3db67a0b20c054be70cf685fd Aug 05 17:28:23.802821 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:23.802804 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:28:24.454462 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:24.454425 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bcf6t" event={"ID":"d0054a8e-de1d-4ae1-989f-a272169a9d9f","Type":"ContainerStarted","Data":"60c1866a264eb11d65fba04a2afe36e8890330c3db67a0b20c054be70cf685fd"} Aug 05 17:28:27.473248 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:27.473217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bcf6t" event={"ID":"d0054a8e-de1d-4ae1-989f-a272169a9d9f","Type":"ContainerStarted","Data":"f180cfac4961190922a5c7bcd7f3a8faf7470249de1b053e38fb01f79f0a2da7"} Aug 05 17:28:27.490382 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:27.490338 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bcf6t" podStartSLOduration=1.6854181110000002 podStartE2EDuration="4.490306738s" totalImagesPullingTime="2.804888627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:28:27.489074939 +0000 UTC m=+364.050360553" watchObservedRunningTime="2026-08-05 17:28:27.490306738 +0000 UTC m=+364.051592351" Aug 05 17:28:50.038178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.038144 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q"] Aug 05 17:28:50.070616 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.070576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q"] Aug 05 17:28:50.070777 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.070666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.075574 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.075549 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 17:28:50.075755 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.075556 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 17:28:50.075755 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.075553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 17:28:50.076116 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.076095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6px7q\"" Aug 05 17:28:50.076212 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.076137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 17:28:50.207548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.207513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzxj\" (UniqueName: \"kubernetes.io/projected/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-kube-api-access-fpzxj\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.207699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.207553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.207699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.207624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-tmp\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.207699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.207643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-manifests\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.207818 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.207715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309135 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzxj\" (UniqueName: \"kubernetes.io/projected/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-kube-api-access-fpzxj\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309135 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309135 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-tmp\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309428 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-manifests\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309428 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.309644 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.309624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-manifests\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.311415 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.311389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-tmp\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.311803 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.311778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.311870 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.311810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.320054 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.320033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzxj\" (UniqueName: \"kubernetes.io/projected/61b6834f-a1f3-4f32-a3f3-ab43e3d614c0-kube-api-access-fpzxj\") pod \"opendatahub-operator-controller-manager-7cc69f54b9-rvn7q\" (UID: \"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0\") " pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.381410 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.381389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:50.531157 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.531129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q"] Aug 05 17:28:50.532312 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:28:50.532276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b6834f_a1f3_4f32_a3f3_ab43e3d614c0.slice/crio-ddfe515fc3f12f77a6d96eda5906a46fb294caf4928eab82fa9b74c74d0ba96d WatchSource:0}: Error finding container ddfe515fc3f12f77a6d96eda5906a46fb294caf4928eab82fa9b74c74d0ba96d: Status 404 returned error can't find the container with id ddfe515fc3f12f77a6d96eda5906a46fb294caf4928eab82fa9b74c74d0ba96d Aug 05 17:28:50.562172 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:50.562115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" event={"ID":"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0","Type":"ContainerStarted","Data":"ddfe515fc3f12f77a6d96eda5906a46fb294caf4928eab82fa9b74c74d0ba96d"} Aug 05 17:28:53.579108 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:53.579077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" event={"ID":"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0","Type":"ContainerStarted","Data":"4c80ed4d37cc2eae1555fe9509e2054a7efd1b0a929041eec1e4a89857438fb0"} Aug 05 17:28:55.589178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:55.589140 2576 generic.go:356] "Generic (PLEG): container finished" podID="61b6834f-a1f3-4f32-a3f3-ab43e3d614c0" containerID="4c80ed4d37cc2eae1555fe9509e2054a7efd1b0a929041eec1e4a89857438fb0" exitCode=0 Aug 05 17:28:55.589632 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:55.589199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" event={"ID":"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0","Type":"ContainerDied","Data":"4c80ed4d37cc2eae1555fe9509e2054a7efd1b0a929041eec1e4a89857438fb0"} Aug 05 17:28:56.595307 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:56.595273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" event={"ID":"61b6834f-a1f3-4f32-a3f3-ab43e3d614c0","Type":"ContainerStarted","Data":"6301673ad335996880d82d2d17cb5920fae6c5553ebba16ae60ea45d3133d222"} Aug 05 17:28:56.595718 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:56.595499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:28:56.626495 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:28:56.626437 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" podStartSLOduration=2.254715299 podStartE2EDuration="6.626415189s" totalImagesPullingTime="3.22696992s" totalInitContainerRuntime="1.14472997s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:28:56.624981803 +0000 UTC m=+393.186267426" watchObservedRunningTime="2026-08-05 17:28:56.626415189 +0000 UTC m=+393.187700803" Aug 05 17:29:07.602165 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:07.602134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cc69f54b9-rvn7q" Aug 05 17:29:09.084120 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.084084 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8"] Aug 05 17:29:09.092462 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.092439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.103937 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.103918 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 17:29:09.108673 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.108650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 17:29:09.109703 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.109672 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:29:09.109828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.109713 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jj7gz\"" Aug 05 17:29:09.109828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.109726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 17:29:09.109828 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.109716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 17:29:09.127593 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.127565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8"] Aug 05 17:29:09.170162 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.170131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-metrics-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.170370 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.170191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfvt\" (UniqueName: \"kubernetes.io/projected/3da1ffa2-0217-4bdd-9864-5f01d55ee104-kube-api-access-kjfvt\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.170370 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.170247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.170370 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.170287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3da1ffa2-0217-4bdd-9864-5f01d55ee104-manager-config\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.271783 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.271734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-metrics-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.271949 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.271799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfvt\" (UniqueName: \"kubernetes.io/projected/3da1ffa2-0217-4bdd-9864-5f01d55ee104-kube-api-access-kjfvt\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.271949 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.271893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.271949 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.271940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3da1ffa2-0217-4bdd-9864-5f01d55ee104-manager-config\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.272601 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.272578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3da1ffa2-0217-4bdd-9864-5f01d55ee104-manager-config\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.274466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.274445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.274466 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.274461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da1ffa2-0217-4bdd-9864-5f01d55ee104-metrics-cert\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.293359 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.293313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfvt\" (UniqueName: \"kubernetes.io/projected/3da1ffa2-0217-4bdd-9864-5f01d55ee104-kube-api-access-kjfvt\") pod \"lws-controller-manager-57fd6d5d59-5nxq8\" (UID: \"3da1ffa2-0217-4bdd-9864-5f01d55ee104\") " pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.401152 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.401053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:09.535021 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.534987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8"] Aug 05 17:29:09.537290 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:29:09.537262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da1ffa2_0217_4bdd_9864_5f01d55ee104.slice/crio-2ec71e73883d846d73db95686ae5fc72159069986e76cf79fee6c24a367d1bf9 WatchSource:0}: Error finding container 2ec71e73883d846d73db95686ae5fc72159069986e76cf79fee6c24a367d1bf9: Status 404 returned error can't find the container with id 2ec71e73883d846d73db95686ae5fc72159069986e76cf79fee6c24a367d1bf9 Aug 05 17:29:09.652747 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:09.652662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" event={"ID":"3da1ffa2-0217-4bdd-9864-5f01d55ee104","Type":"ContainerStarted","Data":"2ec71e73883d846d73db95686ae5fc72159069986e76cf79fee6c24a367d1bf9"} Aug 05 17:29:10.044180 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.044143 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp"] Aug 05 17:29:10.048636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.048610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.051026 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.051002 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 05 17:29:10.051156 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.051028 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 05 17:29:10.051859 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.051837 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jtrt9\"" Aug 05 17:29:10.060217 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.060151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp"] Aug 05 17:29:10.079090 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.079063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmhm\" (UniqueName: \"kubernetes.io/projected/aefb29d2-b574-40fb-82ab-e833035c57bb-kube-api-access-ssmhm\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.079243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.079145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aefb29d2-b574-40fb-82ab-e833035c57bb-tmp\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.079243 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.079176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.180525 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.180484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmhm\" (UniqueName: \"kubernetes.io/projected/aefb29d2-b574-40fb-82ab-e833035c57bb-kube-api-access-ssmhm\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.180909 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.180562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aefb29d2-b574-40fb-82ab-e833035c57bb-tmp\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.180909 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.180589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.180909 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:29:10.180684 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Aug 05 17:29:10.180909 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:29:10.180738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs podName:aefb29d2-b574-40fb-82ab-e833035c57bb nodeName:}" failed. No retries permitted until 2026-08-05 17:29:10.680722279 +0000 UTC m=+407.242007871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs") pod "kube-auth-proxy-746fd96bc4-5s4vp" (UID: "aefb29d2-b574-40fb-82ab-e833035c57bb") : secret "kube-auth-proxy-tls" not found Aug 05 17:29:10.182743 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.182725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aefb29d2-b574-40fb-82ab-e833035c57bb-tmp\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.188650 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.188624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmhm\" (UniqueName: \"kubernetes.io/projected/aefb29d2-b574-40fb-82ab-e833035c57bb-kube-api-access-ssmhm\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.684749 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.684711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.687807 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.687779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aefb29d2-b574-40fb-82ab-e833035c57bb-tls-certs\") pod \"kube-auth-proxy-746fd96bc4-5s4vp\" (UID: \"aefb29d2-b574-40fb-82ab-e833035c57bb\") " pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:10.960457 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:10.960415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" Aug 05 17:29:11.149410 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:11.149367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp"] Aug 05 17:29:11.152751 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:29:11.152703 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefb29d2_b574_40fb_82ab_e833035c57bb.slice/crio-3fa6d3a139721fbd5e94238106ba53b08710807a1e2fe838a2b2fb3e318ff015 WatchSource:0}: Error finding container 3fa6d3a139721fbd5e94238106ba53b08710807a1e2fe838a2b2fb3e318ff015: Status 404 returned error can't find the container with id 3fa6d3a139721fbd5e94238106ba53b08710807a1e2fe838a2b2fb3e318ff015 Aug 05 17:29:11.665133 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:11.665098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" event={"ID":"aefb29d2-b574-40fb-82ab-e833035c57bb","Type":"ContainerStarted","Data":"3fa6d3a139721fbd5e94238106ba53b08710807a1e2fe838a2b2fb3e318ff015"} Aug 05 17:29:12.673460 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:12.673409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" event={"ID":"3da1ffa2-0217-4bdd-9864-5f01d55ee104","Type":"ContainerStarted","Data":"033f7a2e97136cee886cb901ecec15cab0d97a9956f32592c42a2113941938a5"} Aug 05 17:29:12.674062 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:12.673563 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:29:12.693670 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:12.693609 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" podStartSLOduration=0.784754891 podStartE2EDuration="3.693588064s" totalImagesPullingTime="2.908833173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:12.690892205 +0000 UTC m=+409.252177822" watchObservedRunningTime="2026-08-05 17:29:12.693588064 +0000 UTC m=+409.254873679" Aug 05 17:29:14.684248 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:14.684209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" event={"ID":"aefb29d2-b574-40fb-82ab-e833035c57bb","Type":"ContainerStarted","Data":"ded8ace001d55362f03717db09aeac5c9b42875608fbefa7745729346eeb969f"} Aug 05 17:29:14.705303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:14.705254 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746fd96bc4-5s4vp" podStartSLOduration=1.280955589 podStartE2EDuration="4.705238816s" totalImagesPullingTime="3.424283227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:14.703984863 +0000 UTC m=+411.265270488" watchObservedRunningTime="2026-08-05 17:29:14.705238816 +0000 UTC m=+411.266524429" Aug 05 17:29:23.680566 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:29:23.680539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57fd6d5d59-5nxq8" Aug 05 17:30:42.207609 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.207574 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv"] Aug 05 17:30:42.210804 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.210784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.219540 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.219514 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 05 17:30:42.219810 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.219794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 05 17:30:42.220448 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.220434 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-795gl\"" Aug 05 17:30:42.220538 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.220488 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 05 17:30:42.220538 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.220529 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 05 17:30:42.226555 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.226530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv"] Aug 05 17:30:42.278808 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.278769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4bbd07-0cd9-4952-a391-950266e993a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.278808 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.278804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c4bbd07-0cd9-4952-a391-950266e993a8-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.279038 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.278909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwpj\" (UniqueName: \"kubernetes.io/projected/9c4bbd07-0cd9-4952-a391-950266e993a8-kube-api-access-nlwpj\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.379486 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.379445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwpj\" (UniqueName: \"kubernetes.io/projected/9c4bbd07-0cd9-4952-a391-950266e993a8-kube-api-access-nlwpj\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.379699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.379547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4bbd07-0cd9-4952-a391-950266e993a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.379699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.379575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c4bbd07-0cd9-4952-a391-950266e993a8-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.380244 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.380218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c4bbd07-0cd9-4952-a391-950266e993a8-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.382061 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.382037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4bbd07-0cd9-4952-a391-950266e993a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.389589 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.389567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwpj\" (UniqueName: \"kubernetes.io/projected/9c4bbd07-0cd9-4952-a391-950266e993a8-kube-api-access-nlwpj\") pod \"kuadrant-console-plugin-598f86468b-f6wcv\" (UID: \"9c4bbd07-0cd9-4952-a391-950266e993a8\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.520540 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.520459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" Aug 05 17:30:42.662879 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:42.662855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv"] Aug 05 17:30:42.665507 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:30:42.665479 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c4bbd07_0cd9_4952_a391_950266e993a8.slice/crio-79e9ba5d1a3bbf278180211ad4ee80bd55d81fd59ab759075c32e39d314b5a9a WatchSource:0}: Error finding container 79e9ba5d1a3bbf278180211ad4ee80bd55d81fd59ab759075c32e39d314b5a9a: Status 404 returned error can't find the container with id 79e9ba5d1a3bbf278180211ad4ee80bd55d81fd59ab759075c32e39d314b5a9a Aug 05 17:30:43.072635 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:43.072601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" event={"ID":"9c4bbd07-0cd9-4952-a391-950266e993a8","Type":"ContainerStarted","Data":"79e9ba5d1a3bbf278180211ad4ee80bd55d81fd59ab759075c32e39d314b5a9a"} Aug 05 17:30:49.104704 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:49.104661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" event={"ID":"9c4bbd07-0cd9-4952-a391-950266e993a8","Type":"ContainerStarted","Data":"e0bcb35a0f60da480f028e5974dce53c051e358881870d96fef1a1e9fa49a319"} Aug 05 17:30:49.121729 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:30:49.121680 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-f6wcv" podStartSLOduration=1.430199734 podStartE2EDuration="7.121662224s" totalImagesPullingTime="5.69146249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:49.11982116 +0000 UTC m=+505.681106774" watchObservedRunningTime="2026-08-05 17:30:49.121662224 +0000 UTC m=+505.682947838" Aug 05 17:31:30.464212 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.464127 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:31:30.484992 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.484955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:31:30.485149 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.485084 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.488754 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.488729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 05 17:31:30.497092 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.497063 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:31:30.511815 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.511788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b522ddfc-4566-4765-bee8-e8c67be975b0-config-file\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.511957 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.511843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nghg\" (UniqueName: \"kubernetes.io/projected/b522ddfc-4566-4765-bee8-e8c67be975b0-kube-api-access-7nghg\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.612344 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.612286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nghg\" (UniqueName: \"kubernetes.io/projected/b522ddfc-4566-4765-bee8-e8c67be975b0-kube-api-access-7nghg\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.612509 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.612409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b522ddfc-4566-4765-bee8-e8c67be975b0-config-file\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.613036 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.613017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b522ddfc-4566-4765-bee8-e8c67be975b0-config-file\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.625877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.625851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nghg\" (UniqueName: \"kubernetes.io/projected/b522ddfc-4566-4765-bee8-e8c67be975b0-kube-api-access-7nghg\") pod \"limitador-limitador-85d9db4d5d-2xtrv\" (UID: \"b522ddfc-4566-4765-bee8-e8c67be975b0\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.795384 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.795294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:30.931157 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:30.931129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:31:30.933599 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:31:30.933571 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb522ddfc_4566_4765_bee8_e8c67be975b0.slice/crio-acfa7d0af106540c95e988a8a9843383ada7d1ef1b3c5b095779d59b864dfc5d WatchSource:0}: Error finding container acfa7d0af106540c95e988a8a9843383ada7d1ef1b3c5b095779d59b864dfc5d: Status 404 returned error can't find the container with id acfa7d0af106540c95e988a8a9843383ada7d1ef1b3c5b095779d59b864dfc5d Aug 05 17:31:31.287275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:31.287237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" event={"ID":"b522ddfc-4566-4765-bee8-e8c67be975b0","Type":"ContainerStarted","Data":"acfa7d0af106540c95e988a8a9843383ada7d1ef1b3c5b095779d59b864dfc5d"} Aug 05 17:31:33.301799 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:33.301765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" event={"ID":"b522ddfc-4566-4765-bee8-e8c67be975b0","Type":"ContainerStarted","Data":"5e6eb98c7596bbe9459a01a7b28947f0816d5a91b3d5074b354ed0866ca76a10"} Aug 05 17:31:33.302167 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:33.301836 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:33.319426 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:33.319376 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" podStartSLOduration=1.09933766 podStartE2EDuration="3.319359652s" totalImagesPullingTime="2.220021992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:31:33.318630823 +0000 UTC m=+549.879916446" watchObservedRunningTime="2026-08-05 17:31:33.319359652 +0000 UTC m=+549.880645266" Aug 05 17:31:44.306652 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:44.306619 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv" Aug 05 17:31:59.882876 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.882840 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-xtqlw"] Aug 05 17:31:59.885740 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.885723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:31:59.889030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.889007 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7shrc\"" Aug 05 17:31:59.889154 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.889057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 05 17:31:59.899550 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.899523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-xtqlw"] Aug 05 17:31:59.964312 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.964271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zz4\" (UniqueName: \"kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:31:59.964522 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:31:59.964421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.017835 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.017799 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-xtqlw"] Aug 05 17:32:00.018113 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:32:00.018092 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j8zz4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-xtqlw" podUID="e17c028b-c52d-40a5-b70a-080b6e650795" Aug 05 17:32:00.065669 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.065622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zz4\" (UniqueName: \"kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.065871 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.065721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.068079 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.068057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.083994 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.083967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zz4\" (UniqueName: \"kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4\") pod \"authorino-687bd45585-xtqlw\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.417953 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.417921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.424394 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.424369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:00.570847 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.570818 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert\") pod \"e17c028b-c52d-40a5-b70a-080b6e650795\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " Aug 05 17:32:00.571037 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.570903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8zz4\" (UniqueName: \"kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4\") pod \"e17c028b-c52d-40a5-b70a-080b6e650795\" (UID: \"e17c028b-c52d-40a5-b70a-080b6e650795\") " Aug 05 17:32:00.572985 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.572948 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4" (OuterVolumeSpecName: "kube-api-access-j8zz4") pod "e17c028b-c52d-40a5-b70a-080b6e650795" (UID: "e17c028b-c52d-40a5-b70a-080b6e650795"). InnerVolumeSpecName "kube-api-access-j8zz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:32:00.573101 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.572995 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e17c028b-c52d-40a5-b70a-080b6e650795" (UID: "e17c028b-c52d-40a5-b70a-080b6e650795"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:32:00.671575 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.671489 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zz4\" (UniqueName: \"kubernetes.io/projected/e17c028b-c52d-40a5-b70a-080b6e650795-kube-api-access-j8zz4\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:32:00.671575 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:00.671517 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e17c028b-c52d-40a5-b70a-080b6e650795-tls-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:32:01.422274 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:01.422241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-xtqlw" Aug 05 17:32:02.259454 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:02.259419 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-xtqlw"] Aug 05 17:32:02.291865 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:02.291834 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-xtqlw"] Aug 05 17:32:03.977838 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:03.977809 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17c028b-c52d-40a5-b70a-080b6e650795" path="/var/lib/kubelet/pods/e17c028b-c52d-40a5-b70a-080b6e650795/volumes" Aug 05 17:32:23.843479 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:23.843452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:32:23.844035 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:23.843524 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:32:44.552030 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:32:44.551990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:33:02.803012 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.802963 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78"] Aug 05 17:33:02.805445 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.805425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.807913 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.807888 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 17:33:02.808036 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.807915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 05 17:33:02.808750 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.808732 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 17:33:02.808890 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.808811 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6zzxp\"" Aug 05 17:33:02.817538 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.817517 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78"] Aug 05 17:33:02.927019 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.926983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.927019 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.927027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.927239 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.927058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9f4\" (UniqueName: \"kubernetes.io/projected/98443830-bc9b-475b-97f2-e546ad595272-kube-api-access-gw9f4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.927239 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.927105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98443830-bc9b-475b-97f2-e546ad595272-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.927239 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.927153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:02.927239 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:02.927187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.027949 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.027913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.027985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9f4\" (UniqueName: \"kubernetes.io/projected/98443830-bc9b-475b-97f2-e546ad595272-kube-api-access-gw9f4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98443830-bc9b-475b-97f2-e546ad595272-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028131 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028445 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028502 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.028502 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.028484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.030381 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.030362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98443830-bc9b-475b-97f2-e546ad595272-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.030636 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.030616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98443830-bc9b-475b-97f2-e546ad595272-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.038249 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.038226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9f4\" (UniqueName: \"kubernetes.io/projected/98443830-bc9b-475b-97f2-e546ad595272-kube-api-access-gw9f4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78\" (UID: \"98443830-bc9b-475b-97f2-e546ad595272\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.117238 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.117153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:03.241268 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.241240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:33:03.256733 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.256708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78"] Aug 05 17:33:03.259244 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:33:03.259211 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98443830_bc9b_475b_97f2_e546ad595272.slice/crio-9db2fd703a0a942bfe774e5bcafc293d7a7d276820e3aa0e782045e28f6db417 WatchSource:0}: Error finding container 9db2fd703a0a942bfe774e5bcafc293d7a7d276820e3aa0e782045e28f6db417: Status 404 returned error can't find the container with id 9db2fd703a0a942bfe774e5bcafc293d7a7d276820e3aa0e782045e28f6db417 Aug 05 17:33:03.712841 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:03.712795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" event={"ID":"98443830-bc9b-475b-97f2-e546ad595272","Type":"ContainerStarted","Data":"9db2fd703a0a942bfe774e5bcafc293d7a7d276820e3aa0e782045e28f6db417"} Aug 05 17:33:05.724689 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:05.724642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" event={"ID":"98443830-bc9b-475b-97f2-e546ad595272","Type":"ContainerStarted","Data":"73c546f53069a5b2101cbbd2311dde7cd2f73bf8731048d2c8cc7fcc356b3394"} Aug 05 17:33:05.751385 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:05.751315 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" podStartSLOduration=2.216791469 podStartE2EDuration="3.751297439s" totalImagesPullingTime="1.53450597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:05.750059617 +0000 UTC m=+642.311345227" watchObservedRunningTime="2026-08-05 17:33:05.751297439 +0000 UTC m=+642.312583045" Aug 05 17:33:13.117894 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:13.117852 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:13.118313 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:13.117921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:13.130435 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:13.130408 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:13.773682 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:13.773651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78" Aug 05 17:33:32.006650 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:32.006617 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:33:48.706704 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:48.706669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:33:55.206713 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:33:55.206675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:34:01.178699 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.178665 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk"] Aug 05 17:34:01.182207 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.182187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.184585 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.184560 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 05 17:34:01.191840 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.191814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk"] Aug 05 17:34:01.259008 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.258965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.259199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.259025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.259199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.259060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhw8r\" (UniqueName: \"kubernetes.io/projected/652b0a35-339d-4dcf-8f98-bacbc1c480db-kube-api-access-vhw8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.259199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.259080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b0a35-339d-4dcf-8f98-bacbc1c480db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.259199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.259103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.259199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.259121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360376 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhw8r\" (UniqueName: \"kubernetes.io/projected/652b0a35-339d-4dcf-8f98-bacbc1c480db-kube-api-access-vhw8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b0a35-339d-4dcf-8f98-bacbc1c480db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360558 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360832 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360746 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360832 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.360922 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.360913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.362846 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.362821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/652b0a35-339d-4dcf-8f98-bacbc1c480db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.363045 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.363026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b0a35-339d-4dcf-8f98-bacbc1c480db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.368487 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.368463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhw8r\" (UniqueName: \"kubernetes.io/projected/652b0a35-339d-4dcf-8f98-bacbc1c480db-kube-api-access-vhw8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-x9jtk\" (UID: \"652b0a35-339d-4dcf-8f98-bacbc1c480db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.493508 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.493464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:01.626560 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.626526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk"] Aug 05 17:34:01.630752 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:34:01.630724 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652b0a35_339d_4dcf_8f98_bacbc1c480db.slice/crio-55d2150415fdc89a945c90b6990f3e0bcbfb5a04be512af9a9120a2095b8e6f7 WatchSource:0}: Error finding container 55d2150415fdc89a945c90b6990f3e0bcbfb5a04be512af9a9120a2095b8e6f7: Status 404 returned error can't find the container with id 55d2150415fdc89a945c90b6990f3e0bcbfb5a04be512af9a9120a2095b8e6f7 Aug 05 17:34:01.632166 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.632143 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:34:01.986146 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.986116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" event={"ID":"652b0a35-339d-4dcf-8f98-bacbc1c480db","Type":"ContainerStarted","Data":"f0e16c86ac8222e90def993c204248add45506a41210773087b485b0a624064f"} Aug 05 17:34:01.986311 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:01.986161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" event={"ID":"652b0a35-339d-4dcf-8f98-bacbc1c480db","Type":"ContainerStarted","Data":"55d2150415fdc89a945c90b6990f3e0bcbfb5a04be512af9a9120a2095b8e6f7"} Aug 05 17:34:02.003851 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:02.003803 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" podStartSLOduration=0.812686815 podStartE2EDuration="1.003787061s" totalImagesPullingTime="191.100246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:02.003681139 +0000 UTC m=+698.564966758" watchObservedRunningTime="2026-08-05 17:34:02.003787061 +0000 UTC m=+698.565072676" Aug 05 17:34:02.604197 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:02.604118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:34:05.669925 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.669892 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv"] Aug 05 17:34:05.673424 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.673402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.675770 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.675753 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 05 17:34:05.683136 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.683112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv"] Aug 05 17:34:05.800924 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.800887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.801106 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.801003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.801106 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.801044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.801106 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.801098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.801224 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.801142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.801224 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.801172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk95p\" (UniqueName: \"kubernetes.io/projected/9942cc94-73b9-4f47-87e2-ed0adbfd949d-kube-api-access-kk95p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk95p\" (UniqueName: \"kubernetes.io/projected/9942cc94-73b9-4f47-87e2-ed0adbfd949d-kube-api-access-kk95p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902482 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902687 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902752 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902804 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.902864 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.902844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.904627 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.904603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9942cc94-73b9-4f47-87e2-ed0adbfd949d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.904849 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.904834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9942cc94-73b9-4f47-87e2-ed0adbfd949d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.910625 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.910600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk95p\" (UniqueName: \"kubernetes.io/projected/9942cc94-73b9-4f47-87e2-ed0adbfd949d-kube-api-access-kk95p\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv\" (UID: \"9942cc94-73b9-4f47-87e2-ed0adbfd949d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:05.983621 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:05.983592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:06.122944 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:06.122918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv"] Aug 05 17:34:06.124656 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:34:06.124629 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9942cc94_73b9_4f47_87e2_ed0adbfd949d.slice/crio-390327f8b22f74738aba9962bf30f05cd00eeea14aa209474eea92631f18df1b WatchSource:0}: Error finding container 390327f8b22f74738aba9962bf30f05cd00eeea14aa209474eea92631f18df1b: Status 404 returned error can't find the container with id 390327f8b22f74738aba9962bf30f05cd00eeea14aa209474eea92631f18df1b Aug 05 17:34:07.011343 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:07.011295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" event={"ID":"9942cc94-73b9-4f47-87e2-ed0adbfd949d","Type":"ContainerStarted","Data":"61119195f09ddd9ef4177100d446b4ac4ac601bdc509ff80a7011defb135f827"} Aug 05 17:34:07.011722 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:07.011364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" event={"ID":"9942cc94-73b9-4f47-87e2-ed0adbfd949d","Type":"ContainerStarted","Data":"390327f8b22f74738aba9962bf30f05cd00eeea14aa209474eea92631f18df1b"} Aug 05 17:34:07.034719 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:07.034665 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" podStartSLOduration=1.8305558039999998 podStartE2EDuration="2.034647904s" totalImagesPullingTime="204.0921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:07.032614908 +0000 UTC m=+703.593900519" watchObservedRunningTime="2026-08-05 17:34:07.034647904 +0000 UTC m=+703.595933518" Aug 05 17:34:11.494130 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:11.494085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:11.494562 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:11.494199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:11.506701 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:11.506677 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:12.045434 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:12.045400 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-x9jtk" Aug 05 17:34:12.910618 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:12.910585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:34:15.984429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:15.984333 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:15.984429 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:15.984397 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:15.997182 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:15.997151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:34:16.064224 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:34:16.064191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv" Aug 05 17:35:40.112175 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:35:40.112136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:35:49.209590 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:35:49.209502 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:36:02.316074 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:36:02.316035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:37:23.870235 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:37:23.870153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:37:23.873083 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:37:23.873057 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:37:40.807647 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:37:40.807610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:37:49.918440 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:37:49.918406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:38:49.915149 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:38:49.915072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:38:56.612750 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:38:56.612719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:12.021120 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:12.021086 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:18.726060 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:18.726017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:28.012744 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:28.012705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:34.607575 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:34.607537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:48.126819 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:48.126782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:53.114177 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:53.114136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:39:58.204190 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:39:58.204148 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:40:05.113945 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:40:05.113911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:40:11.910743 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:40:11.910704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:40:23.313128 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:40:23.313041 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:40:34.008872 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:40:34.008834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:40:57.509714 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:40:57.509681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:08.002993 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:08.002961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:13.016866 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:13.016824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:20.835359 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:20.835313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:28.208921 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:28.208880 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:33.411893 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:33.411862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:39.198912 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:39.198879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:46.798819 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:46.798742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:41:58.300991 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:41:58.300959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:07.408275 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:07.408243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:15.603025 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:15.602986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:23.904452 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:23.904420 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:42:23.909787 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:23.909767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:42:25.903208 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:25.903175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:36.514472 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:36.514433 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:41.697206 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:41.697171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:42:52.315960 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:42:52.315925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:45:10.303335 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:45:10.303281 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:45:15.744012 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:45:15.743974 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:45:41.202757 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:45:41.202721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:15.104931 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:15.104901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:21.407028 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:21.406993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:30.802563 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:30.802526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:36.001753 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:36.001723 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:46.412573 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:46.412527 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:46:53.025101 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:46:53.025058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:01.605186 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:01.605149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:06.913996 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:06.913959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:17.623837 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:17.623802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:23.931844 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:23.931811 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:47:23.941981 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:23.941946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:47:27.808214 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:27.808176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:31.959967 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:31.959930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:47:56.709418 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:47:56.709336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:48:02.004919 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:02.004881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:48:08.764833 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:08.764791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:48:13.968479 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:13.968433 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:48:13.974611 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:13.974582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:13.976792 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:13.976768 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-836d574c-kserve-self-signed-certs\"" Aug 05 17:48:13.982887 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:13.982856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:48:14.051161 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6l7m\" (UniqueName: \"kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.051161 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.051473 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.051473 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.051473 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.051473 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.051399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.152924 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.152879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.152975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6l7m\" (UniqueName: \"kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153141 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153301 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153301 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153560 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153560 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.153717 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.153593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.155366 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.155345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.155530 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.155515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.162972 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.162934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6l7m\" (UniqueName: \"kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m\") pod \"e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.286868 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.286780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:14.426534 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.426508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:48:14.428533 ip-10-0-131-14 kubenswrapper[2576]: W0805 17:48:14.428505 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0c32de0_b105_401a_84c9_a221656ca9f6.slice/crio-8b070ebdf9a5c2cfdfc2e594856d5c0a3222c3ef753caf3caf82cf97e2d803bd WatchSource:0}: Error finding container 8b070ebdf9a5c2cfdfc2e594856d5c0a3222c3ef753caf3caf82cf97e2d803bd: Status 404 returned error can't find the container with id 8b070ebdf9a5c2cfdfc2e594856d5c0a3222c3ef753caf3caf82cf97e2d803bd Aug 05 17:48:14.863234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.863197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" event={"ID":"e0c32de0-b105-401a-84c9-a221656ca9f6","Type":"ContainerStarted","Data":"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88"} Aug 05 17:48:14.863234 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.863239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" event={"ID":"e0c32de0-b105-401a-84c9-a221656ca9f6","Type":"ContainerStarted","Data":"8b070ebdf9a5c2cfdfc2e594856d5c0a3222c3ef753caf3caf82cf97e2d803bd"} Aug 05 17:48:14.881701 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:14.881641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" podStartSLOduration=1.8816191820000001 podStartE2EDuration="1.881619182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:48:14.880883066 +0000 UTC m=+1551.442168703" watchObservedRunningTime="2026-08-05 17:48:14.881619182 +0000 UTC m=+1551.442904796" Aug 05 17:48:18.216922 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:18.216877 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:48:24.286950 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:24.286912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:24.287531 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:24.286966 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:24.299816 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:24.299786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:24.924044 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:24.924009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:48:50.998744 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:50.998707 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:48:51.402184 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:51.402094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:48:59.002242 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:48:59.002206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:49:06.018877 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.018840 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" podUID="e0c32de0-b105-401a-84c9-a221656ca9f6" containerName="main" containerID="cri-o://2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88" gracePeriod=45 Aug 05 17:49:06.272986 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.272918 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:49:06.339242 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339190 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339242 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6l7m\" (UniqueName: \"kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339535 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339535 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339535 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339535 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home\") pod \"e0c32de0-b105-401a-84c9-a221656ca9f6\" (UID: \"e0c32de0-b105-401a-84c9-a221656ca9f6\") " Aug 05 17:49:06.339741 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339604 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:49:06.339741 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339629 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:49:06.339741 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339702 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:06.339845 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.339742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home" (OuterVolumeSpecName: "home") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:49:06.341551 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.341525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm" (OuterVolumeSpecName: "dshm") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:49:06.341656 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.341525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:49:06.341656 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.341560 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m" (OuterVolumeSpecName: "kube-api-access-p6l7m") pod "e0c32de0-b105-401a-84c9-a221656ca9f6" (UID: "e0c32de0-b105-401a-84c9-a221656ca9f6"). InnerVolumeSpecName "kube-api-access-p6l7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:49:06.440476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.440426 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c32de0-b105-401a-84c9-a221656ca9f6-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:06.440476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.440470 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6l7m\" (UniqueName: \"kubernetes.io/projected/e0c32de0-b105-401a-84c9-a221656ca9f6-kube-api-access-p6l7m\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:06.440476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.440479 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:06.440476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.440488 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:06.440476 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:06.440497 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c32de0-b105-401a-84c9-a221656ca9f6-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 17:49:07.124442 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.124403 2576 generic.go:356] "Generic (PLEG): container finished" podID="e0c32de0-b105-401a-84c9-a221656ca9f6" containerID="2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88" exitCode=0 Aug 05 17:49:07.124442 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.124446 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" Aug 05 17:49:07.124896 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.124482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" event={"ID":"e0c32de0-b105-401a-84c9-a221656ca9f6","Type":"ContainerDied","Data":"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88"} Aug 05 17:49:07.124896 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.124517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2" event={"ID":"e0c32de0-b105-401a-84c9-a221656ca9f6","Type":"ContainerDied","Data":"8b070ebdf9a5c2cfdfc2e594856d5c0a3222c3ef753caf3caf82cf97e2d803bd"} Aug 05 17:49:07.124896 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.124537 2576 scope.go:117] "RemoveContainer" containerID="2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88" Aug 05 17:49:07.133392 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.133371 2576 scope.go:117] "RemoveContainer" containerID="2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88" Aug 05 17:49:07.133668 ip-10-0-131-14 kubenswrapper[2576]: E0805 17:49:07.133649 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88\": container with ID starting with 2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88 not found: ID does not exist" containerID="2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88" Aug 05 17:49:07.133749 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.133680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88"} err="failed to get container status \"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88\": rpc error: code = NotFound desc = could not find container \"2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88\": container with ID starting with 2a883e424cb8a4da5a6f6a2a8e9470a2401cad49fdf059a7aae8343a13077b88 not found: ID does not exist" Aug 05 17:49:07.145548 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.145518 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:49:07.149220 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.149200 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-836d574c-kserve-6c5bb9cff5-6clh2"] Aug 05 17:49:07.977927 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:07.977896 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c32de0-b105-401a-84c9-a221656ca9f6" path="/var/lib/kubelet/pods/e0c32de0-b105-401a-84c9-a221656ca9f6/volumes" Aug 05 17:49:08.303630 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:08.303544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:49:17.913763 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:17.913726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:49:36.099178 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:36.099141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:49:53.403923 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:49:53.403888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:00.108071 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:00.108038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:12.110587 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:12.110552 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:17.311720 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:17.311677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:35.011896 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:35.011861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:40.111117 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:40.111083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:53.811728 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:53.811650 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:50:59.100375 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:50:59.100338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:05.313052 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:05.313016 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:09.415768 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:09.415738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:17.004997 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:17.004963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:25.998425 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:25.998387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:33.213655 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:33.213619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:43.007569 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:43.007535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:50.100752 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:50.100719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:51:59.713072 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:51:59.713033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:15.124889 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:15.124857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:22.002303 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:22.002272 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:23.966092 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:23.966060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:52:23.972672 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:23.972645 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:52:32.613046 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:32.613008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:38.948244 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:38.948210 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:48.119536 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:48.119500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:52:57.913066 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:52:57.913027 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:53:08.126943 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:53:08.126908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:53:38.516511 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:53:38.516479 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:53:45.622603 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:53:45.622519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:54:18.003301 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:54:18.003264 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:54:26.015199 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:54:26.015167 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:54:53.505495 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:54:53.505460 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:55:03.099156 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:55:03.099120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:55:29.406090 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:55:29.405999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:55:42.203056 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:55:42.203017 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:57:16.304110 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:57:16.304031 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:57:23.931539 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:57:23.931500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:57:23.999092 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:57:23.999064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:57:24.007175 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:57:24.007154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 17:57:38.008895 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:57:38.008858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:58:18.309723 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:58:18.309645 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 17:58:57.205677 ip-10-0-131-14 kubenswrapper[2576]: I0805 17:58:57.205638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:00:41.107698 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:00:41.107664 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:00:49.917292 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:00:49.917256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:01:17.302963 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:01:17.302883 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:01:48.337725 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:01:48.337686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:02:03.424774 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:02:03.424738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:02:09.105819 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:02:09.105784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:02:24.027625 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:02:24.027597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:02:24.046294 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:02:24.046269 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:02:59.901936 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:02:59.901842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:03:12.204276 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:03:12.204237 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:03:19.707760 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:03:19.707721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:03:27.212734 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:03:27.212695 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:04:13.107186 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:04:13.107147 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:04:40.624234 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:04:40.624097 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:05:08.393906 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.393872 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:05:08.394449 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.394224 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c32de0-b105-401a-84c9-a221656ca9f6" containerName="main" Aug 05 18:05:08.394449 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.394235 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c32de0-b105-401a-84c9-a221656ca9f6" containerName="main" Aug 05 18:05:08.394449 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.394313 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c32de0-b105-401a-84c9-a221656ca9f6" containerName="main" Aug 05 18:05:08.397417 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.397398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.400029 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.400010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83d598\"/\"default-dockercfg-q46gv\"" Aug 05 18:05:08.400935 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.400894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83d598\"/\"openshift-service-ca.crt\"" Aug 05 18:05:08.401052 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.400948 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83d598\"/\"kube-root-ca.crt\"" Aug 05 18:05:08.401052 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.400961 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83d598\"/\"test-model-83d598-kserve-self-signed-certs\"" Aug 05 18:05:08.407490 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.407468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:05:08.563346 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.563540 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.563540 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptz7l\" (UniqueName: \"kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.563540 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.563540 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.563678 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.563556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.664730 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.664730 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptz7l\" (UniqueName: \"kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.664927 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.664927 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.664927 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.665040 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.664952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.665097 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.665077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.665183 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.665165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.665224 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.665202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.667182 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.667150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.667380 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.667362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.675914 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.675889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptz7l\" (UniqueName: \"kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l\") pod \"test-model-83d598-kserve-6cc454fd7-f6dh4\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.707359 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.707308 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:08.851331 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:08.851295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:05:08.853408 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:05:08.853369 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc679ecef_b831_438d_8cd1_66d544bf3b02.slice/crio-e9f0ad3a067efb23972d240ffa707f0fc81ad91fffeb16eb9b4eb1d244bbf02a WatchSource:0}: Error finding container e9f0ad3a067efb23972d240ffa707f0fc81ad91fffeb16eb9b4eb1d244bbf02a: Status 404 returned error can't find the container with id e9f0ad3a067efb23972d240ffa707f0fc81ad91fffeb16eb9b4eb1d244bbf02a Aug 05 18:05:09.503080 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:09.503050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" event={"ID":"c679ecef-b831-438d-8cd1-66d544bf3b02","Type":"ContainerStarted","Data":"74aea0028c57f1ce70b0499458f19fd784c13c83011b1a143870ae63d389cd1b"} Aug 05 18:05:09.503629 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:09.503088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" event={"ID":"c679ecef-b831-438d-8cd1-66d544bf3b02","Type":"ContainerStarted","Data":"e9f0ad3a067efb23972d240ffa707f0fc81ad91fffeb16eb9b4eb1d244bbf02a"} Aug 05 18:05:09.526329 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:09.526265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" podStartSLOduration=1.526247133 podStartE2EDuration="1.526247133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:05:09.524490007 +0000 UTC m=+2566.085775625" watchObservedRunningTime="2026-08-05 18:05:09.526247133 +0000 UTC m=+2566.087532747" Aug 05 18:05:10.814398 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:10.814364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:05:18.708085 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:18.708048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:18.708655 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:18.708102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:18.720578 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:18.720550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:19.561558 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:19.561524 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:05:19.914430 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:19.914342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:05:57.008707 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:05:57.008624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:06:36.788107 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:36.788070 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:06:40.324525 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:40.324490 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:06:50.615304 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:50.615266 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:06:51.806818 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:51.806777 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" podUID="c679ecef-b831-438d-8cd1-66d544bf3b02" containerName="main" containerID="cri-o://74aea0028c57f1ce70b0499458f19fd784c13c83011b1a143870ae63d389cd1b" gracePeriod=45 Aug 05 18:06:51.980781 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:51.980735 2576 generic.go:356] "Generic (PLEG): container finished" podID="c679ecef-b831-438d-8cd1-66d544bf3b02" containerID="74aea0028c57f1ce70b0499458f19fd784c13c83011b1a143870ae63d389cd1b" exitCode=0 Aug 05 18:06:51.980925 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:51.980800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" event={"ID":"c679ecef-b831-438d-8cd1-66d544bf3b02","Type":"ContainerDied","Data":"74aea0028c57f1ce70b0499458f19fd784c13c83011b1a143870ae63d389cd1b"} Aug 05 18:06:52.057924 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.057863 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:06:52.130853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.130823 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.130853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.130861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptz7l\" (UniqueName: \"kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.131087 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.130883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.131087 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.130981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.131087 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.131028 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.131087 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.131066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs\") pod \"c679ecef-b831-438d-8cd1-66d544bf3b02\" (UID: \"c679ecef-b831-438d-8cd1-66d544bf3b02\") " Aug 05 18:06:52.131298 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.131236 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache" (OuterVolumeSpecName: "model-cache") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:52.131372 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.131300 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:52.132673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.131220 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home" (OuterVolumeSpecName: "home") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:52.132673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.132031 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.132673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.132054 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.132673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.132067 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.133907 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.133850 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:06:52.133907 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.133857 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm" (OuterVolumeSpecName: "dshm") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:52.133907 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.133883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l" (OuterVolumeSpecName: "kube-api-access-ptz7l") pod "c679ecef-b831-438d-8cd1-66d544bf3b02" (UID: "c679ecef-b831-438d-8cd1-66d544bf3b02"). InnerVolumeSpecName "kube-api-access-ptz7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:06:52.232847 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.232807 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679ecef-b831-438d-8cd1-66d544bf3b02-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.232847 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.232842 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679ecef-b831-438d-8cd1-66d544bf3b02-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.232847 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.232852 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptz7l\" (UniqueName: \"kubernetes.io/projected/c679ecef-b831-438d-8cd1-66d544bf3b02-kube-api-access-ptz7l\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:06:52.987638 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.987603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" event={"ID":"c679ecef-b831-438d-8cd1-66d544bf3b02","Type":"ContainerDied","Data":"e9f0ad3a067efb23972d240ffa707f0fc81ad91fffeb16eb9b4eb1d244bbf02a"} Aug 05 18:06:52.987638 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.987639 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4" Aug 05 18:06:52.988139 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:52.987663 2576 scope.go:117] "RemoveContainer" containerID="74aea0028c57f1ce70b0499458f19fd784c13c83011b1a143870ae63d389cd1b" Aug 05 18:06:53.008782 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:53.008756 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:06:53.012489 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:53.012466 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-83d598/test-model-83d598-kserve-6cc454fd7-f6dh4"] Aug 05 18:06:53.984592 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:06:53.984556 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c679ecef-b831-438d-8cd1-66d544bf3b02" path="/var/lib/kubelet/pods/c679ecef-b831-438d-8cd1-66d544bf3b02/volumes" Aug 05 18:07:08.308831 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:07:08.308789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:07:20.935436 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:07:20.935336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:07:24.063241 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:07:24.063205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:07:24.075200 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:07:24.075178 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:07:49.609979 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:07:49.609942 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:08:31.509449 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:08:31.509417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:08:53.310795 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:08:53.310706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:09:07.313115 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:09:07.313078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:09:35.426141 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:09:35.426094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:10:01.982395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.982366 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:10:01.984771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.982687 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c679ecef-b831-438d-8cd1-66d544bf3b02" containerName="main" Aug 05 18:10:01.984771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.982698 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c679ecef-b831-438d-8cd1-66d544bf3b02" containerName="main" Aug 05 18:10:01.984771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.982772 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c679ecef-b831-438d-8cd1-66d544bf3b02" containerName="main" Aug 05 18:10:01.985512 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.985497 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:01.992575 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.992553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"default-dockercfg-xvpqv\"" Aug 05 18:10:01.993510 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.993492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"kube-root-ca.crt\"" Aug 05 18:10:01.993569 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.993526 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"auth-test-model-ba69fc-kserve-self-signed-certs\"" Aug 05 18:10:01.993607 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:01.993492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"openshift-service-ca.crt\"" Aug 05 18:10:02.008390 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.008357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:10:02.075124 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ftl\" (UniqueName: \"kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.075276 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.075405 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.075483 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.075483 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.075637 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.075543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.176972 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.176938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177115 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ftl\" (UniqueName: \"kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177115 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177115 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177255 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177255 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177535 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.177593 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.177574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.179239 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.179219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.179421 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.179403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.186451 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.186430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ftl\" (UniqueName: \"kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl\") pod \"auth-test-model-ba69fc-kserve-865bc94f59-vj7xx\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.294420 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.294351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:02.425131 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.425107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:10:02.426782 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:10:02.426756 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe1e0ba1_8f00_4f13_93ad_971093491fb2.slice/crio-1d83b2dfe83cab936252dce80bdb30458c08848e3d20e6d16eac6be8a46c695b WatchSource:0}: Error finding container 1d83b2dfe83cab936252dce80bdb30458c08848e3d20e6d16eac6be8a46c695b: Status 404 returned error can't find the container with id 1d83b2dfe83cab936252dce80bdb30458c08848e3d20e6d16eac6be8a46c695b Aug 05 18:10:02.886109 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.886069 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" event={"ID":"fe1e0ba1-8f00-4f13-93ad-971093491fb2","Type":"ContainerStarted","Data":"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc"} Aug 05 18:10:02.886284 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.886118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" event={"ID":"fe1e0ba1-8f00-4f13-93ad-971093491fb2","Type":"ContainerStarted","Data":"1d83b2dfe83cab936252dce80bdb30458c08848e3d20e6d16eac6be8a46c695b"} Aug 05 18:10:02.904984 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:02.904934 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" podStartSLOduration=1.904918758 podStartE2EDuration="1.904918758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:10:02.903486317 +0000 UTC m=+2859.464771931" watchObservedRunningTime="2026-08-05 18:10:02.904918758 +0000 UTC m=+2859.466204423" Aug 05 18:10:04.314191 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:04.314155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:10:12.295412 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:12.295371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:12.295412 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:12.295418 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:12.307916 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:12.307890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:12.950486 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:12.950450 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:10:21.654046 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.653965 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:10:21.657814 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.657785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.661168 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.661136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"default-dockercfg-6dppk\"" Aug 05 18:10:21.661168 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.661161 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"openshift-service-ca.crt\"" Aug 05 18:10:21.661392 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.661194 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"kube-root-ca.crt\"" Aug 05 18:10:21.661392 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.661194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"auth-test-model-594b65-kserve-self-signed-certs\"" Aug 05 18:10:21.666614 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.666472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:10:21.763952 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.763916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.764129 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.763983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.764129 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.764001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.764129 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.764058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.764129 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.764099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbh9\" (UniqueName: \"kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.764278 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.764135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865408 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865408 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbh9\" (UniqueName: \"kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865673 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865835 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865892 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865945 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.865982 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.865948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.867719 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.867697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.868035 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.868015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.873526 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.873505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbh9\" (UniqueName: \"kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9\") pod \"auth-test-model-594b65-kserve-59d997bd99-v9qgr\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:21.969484 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:21.969450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:22.105419 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:22.105389 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:10:22.108972 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:10:22.108933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc48cbe_14f9_41ff_a893_152e83a0cb5e.slice/crio-80eed1f04b1b5dd19245d6a5896370e1fcb3df608d90746a74442fe3c243b45e WatchSource:0}: Error finding container 80eed1f04b1b5dd19245d6a5896370e1fcb3df608d90746a74442fe3c243b45e: Status 404 returned error can't find the container with id 80eed1f04b1b5dd19245d6a5896370e1fcb3df608d90746a74442fe3c243b45e Aug 05 18:10:22.986573 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:22.986536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" event={"ID":"9fc48cbe-14f9-41ff-a893-152e83a0cb5e","Type":"ContainerStarted","Data":"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378"} Aug 05 18:10:22.986573 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:22.986575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" event={"ID":"9fc48cbe-14f9-41ff-a893-152e83a0cb5e","Type":"ContainerStarted","Data":"80eed1f04b1b5dd19245d6a5896370e1fcb3df608d90746a74442fe3c243b45e"} Aug 05 18:10:23.007016 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:23.006957 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" podStartSLOduration=2.006939779 podStartE2EDuration="2.006939779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:10:23.0046842 +0000 UTC m=+2879.565969819" watchObservedRunningTime="2026-08-05 18:10:23.006939779 +0000 UTC m=+2879.568225393" Aug 05 18:10:24.315591 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:24.315555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:10:31.970132 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:31.970089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:31.970562 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:31.970147 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:31.982745 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:31.982721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:32.041416 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:32.041385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:10:46.613240 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:10:46.613199 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:11:01.911786 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:11:01.911751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:11:13.909861 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:11:13.909824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:11:28.510387 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:11:28.510349 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:11:40.413452 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:11:40.413410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:11:55.102954 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:11:55.102869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:12:06.609716 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:06.609678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:12:21.299415 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:21.299374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:12:24.092896 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:24.092859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:12:24.106747 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:24.106722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:12:33.444869 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:33.444831 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:12:47.808500 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:47.808464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:12:59.513261 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:12:59.513217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:13:14.204588 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:14.204549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:13:18.258146 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:18.258059 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:13:19.164010 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:19.163969 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:13:23.915847 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:23.915814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:13:33.279778 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.279715 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" podUID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" containerName="main" containerID="cri-o://628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc" gracePeriod=45 Aug 05 18:13:33.533490 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.533419 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:13:33.640114 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ftl\" (UniqueName: \"kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640114 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640119 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640229 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir\") pod \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\" (UID: \"fe1e0ba1-8f00-4f13-93ad-971093491fb2\") " Aug 05 18:13:33.640598 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640454 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:33.640652 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home" (OuterVolumeSpecName: "home") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:33.640691 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.640633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:33.642373 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.642348 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm" (OuterVolumeSpecName: "dshm") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:33.642475 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.642433 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:13:33.642529 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.642508 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl" (OuterVolumeSpecName: "kube-api-access-m7ftl") pod "fe1e0ba1-8f00-4f13-93ad-971093491fb2" (UID: "fe1e0ba1-8f00-4f13-93ad-971093491fb2"). InnerVolumeSpecName "kube-api-access-m7ftl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:33.741646 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741617 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.741646 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741642 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ftl\" (UniqueName: \"kubernetes.io/projected/fe1e0ba1-8f00-4f13-93ad-971093491fb2-kube-api-access-m7ftl\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.741646 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741652 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.741861 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741661 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e0ba1-8f00-4f13-93ad-971093491fb2-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.741861 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741670 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.741861 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.741678 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e0ba1-8f00-4f13-93ad-971093491fb2-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:33.886572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.886478 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" containerID="628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc" exitCode=0 Aug 05 18:13:33.886572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.886514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" event={"ID":"fe1e0ba1-8f00-4f13-93ad-971093491fb2","Type":"ContainerDied","Data":"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc"} Aug 05 18:13:33.886572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.886554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" event={"ID":"fe1e0ba1-8f00-4f13-93ad-971093491fb2","Type":"ContainerDied","Data":"1d83b2dfe83cab936252dce80bdb30458c08848e3d20e6d16eac6be8a46c695b"} Aug 05 18:13:33.886572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.886573 2576 scope.go:117] "RemoveContainer" containerID="628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc" Aug 05 18:13:33.886864 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.886526 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx" Aug 05 18:13:33.895895 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.895876 2576 scope.go:117] "RemoveContainer" containerID="628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc" Aug 05 18:13:33.896148 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:13:33.896130 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc\": container with ID starting with 628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc not found: ID does not exist" containerID="628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc" Aug 05 18:13:33.896226 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.896156 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc"} err="failed to get container status \"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc\": rpc error: code = NotFound desc = could not find container \"628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc\": container with ID starting with 628e30659362f033fdb759d1888eae3d2aee78838c93c6dbb4d2cd6b047fa1dc not found: ID does not exist" Aug 05 18:13:33.907578 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.907557 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:13:33.913592 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.913571 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/auth-test-model-ba69fc-kserve-865bc94f59-vj7xx"] Aug 05 18:13:33.978549 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:33.978526 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" path="/var/lib/kubelet/pods/fe1e0ba1-8f00-4f13-93ad-971093491fb2/volumes" Aug 05 18:13:34.183650 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.183564 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" podUID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" containerName="main" containerID="cri-o://7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378" gracePeriod=45 Aug 05 18:13:34.431288 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.431265 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:13:34.549517 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549723 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549723 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549547 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clbh9\" (UniqueName: \"kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549723 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549598 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549723 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549723 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549639 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs\") pod \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\" (UID: \"9fc48cbe-14f9-41ff-a893-152e83a0cb5e\") " Aug 05 18:13:34.549996 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549795 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home" (OuterVolumeSpecName: "home") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:34.549996 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:34.549996 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549891 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:34.549996 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549982 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.549996 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.549995 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.550191 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.550004 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.551776 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.551754 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm" (OuterVolumeSpecName: "dshm") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:34.551776 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.551755 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9" (OuterVolumeSpecName: "kube-api-access-clbh9") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "kube-api-access-clbh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:34.551898 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.551774 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fc48cbe-14f9-41ff-a893-152e83a0cb5e" (UID: "9fc48cbe-14f9-41ff-a893-152e83a0cb5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:13:34.651199 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.651166 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.651199 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.651194 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.651425 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.651210 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbh9\" (UniqueName: \"kubernetes.io/projected/9fc48cbe-14f9-41ff-a893-152e83a0cb5e-kube-api-access-clbh9\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:13:34.893619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.893528 2576 generic.go:356] "Generic (PLEG): container finished" podID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" containerID="7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378" exitCode=0 Aug 05 18:13:34.893619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.893565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" event={"ID":"9fc48cbe-14f9-41ff-a893-152e83a0cb5e","Type":"ContainerDied","Data":"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378"} Aug 05 18:13:34.893619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.893572 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" Aug 05 18:13:34.893619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.893592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr" event={"ID":"9fc48cbe-14f9-41ff-a893-152e83a0cb5e","Type":"ContainerDied","Data":"80eed1f04b1b5dd19245d6a5896370e1fcb3df608d90746a74442fe3c243b45e"} Aug 05 18:13:34.893619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.893610 2576 scope.go:117] "RemoveContainer" containerID="7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378" Aug 05 18:13:34.904626 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.904267 2576 scope.go:117] "RemoveContainer" containerID="7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378" Aug 05 18:13:34.904824 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:13:34.904796 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378\": container with ID starting with 7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378 not found: ID does not exist" containerID="7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378" Aug 05 18:13:34.904905 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.904834 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378"} err="failed to get container status \"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378\": rpc error: code = NotFound desc = could not find container \"7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378\": container with ID starting with 7a4d221317510c00c0a22dcb4cf6ca1ca9593c072504a49d605c4b18e2f8a378 not found: ID does not exist" Aug 05 18:13:34.919722 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.919699 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:13:34.923527 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:34.923506 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/auth-test-model-594b65-kserve-59d997bd99-v9qgr"] Aug 05 18:13:35.978096 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:35.978062 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" path="/var/lib/kubelet/pods/9fc48cbe-14f9-41ff-a893-152e83a0cb5e/volumes" Aug 05 18:13:36.075230 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075200 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:13:36.075562 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075548 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" containerName="main" Aug 05 18:13:36.075607 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075564 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" containerName="main" Aug 05 18:13:36.075607 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075587 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" containerName="main" Aug 05 18:13:36.075607 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075592 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" containerName="main" Aug 05 18:13:36.075698 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075644 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1e0ba1-8f00-4f13-93ad-971093491fb2" containerName="main" Aug 05 18:13:36.075698 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.075653 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc48cbe-14f9-41ff-a893-152e83a0cb5e" containerName="main" Aug 05 18:13:36.080015 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.080001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.083309 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.083289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"default-dockercfg-xvpqv\"" Aug 05 18:13:36.083309 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.083299 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"sub-test-model-ba69fc-kserve-self-signed-certs\"" Aug 05 18:13:36.083506 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.083298 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"kube-root-ca.crt\"" Aug 05 18:13:36.083506 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.083373 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ba69fc\"/\"openshift-service-ca.crt\"" Aug 05 18:13:36.086750 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.086728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:13:36.164703 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.164868 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.164868 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.164868 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.164868 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9vv\" (UniqueName: \"kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.165008 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.164882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266094 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266094 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266094 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266094 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9vv\" (UniqueName: \"kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266454 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266454 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266553 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266553 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.266633 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.266587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.268424 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.268400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.268660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.268645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.273519 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.273495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9vv\" (UniqueName: \"kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv\") pod \"sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.390736 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.390705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:36.521938 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.521914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:13:36.524128 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:13:36.524097 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6205f1ae_7f25_4f7c_b219_f5db161b83f2.slice/crio-69c0f95a56b6c1579f114958cc83fcb2a84ba9cafb888f6eca8e5cd16265d825 WatchSource:0}: Error finding container 69c0f95a56b6c1579f114958cc83fcb2a84ba9cafb888f6eca8e5cd16265d825: Status 404 returned error can't find the container with id 69c0f95a56b6c1579f114958cc83fcb2a84ba9cafb888f6eca8e5cd16265d825 Aug 05 18:13:36.906001 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.905907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" event={"ID":"6205f1ae-7f25-4f7c-b219-f5db161b83f2","Type":"ContainerStarted","Data":"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0"} Aug 05 18:13:36.906001 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.905958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" event={"ID":"6205f1ae-7f25-4f7c-b219-f5db161b83f2","Type":"ContainerStarted","Data":"69c0f95a56b6c1579f114958cc83fcb2a84ba9cafb888f6eca8e5cd16265d825"} Aug 05 18:13:36.922874 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:36.922820 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" podStartSLOduration=0.922802759 podStartE2EDuration="922.802759ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:13:36.922378005 +0000 UTC m=+3073.483663618" watchObservedRunningTime="2026-08-05 18:13:36.922802759 +0000 UTC m=+3073.484088372" Aug 05 18:13:38.609434 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:38.609398 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:13:46.391118 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:46.391084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:46.391580 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:46.391169 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:46.403729 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:46.403707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:13:46.963978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:13:46.963941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:14:15.084882 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.084844 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:14:15.088541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.088508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.091156 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.091132 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"sub-test-model-594b65-kserve-self-signed-certs\"" Aug 05 18:14:15.092022 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.091998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"kube-root-ca.crt\"" Aug 05 18:14:15.092144 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.092075 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"default-dockercfg-6dppk\"" Aug 05 18:14:15.092144 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.092101 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"openshift-service-ca.crt\"" Aug 05 18:14:15.100684 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.100657 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:14:15.118437 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.118437 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.118437 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ddp\" (UniqueName: \"kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.118437 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.118994 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.118994 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.118696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219276 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219276 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ddp\" (UniqueName: \"kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219793 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219793 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.219924 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.219852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.221726 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.221690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.221965 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.221946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.228584 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.228563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ddp\" (UniqueName: \"kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp\") pod \"sub-test-model-594b65-kserve-578797979c-9qn9b\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.399793 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.399678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:15.540560 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:15.540534 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:14:15.543280 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:14:15.543246 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fbcf7c5_0700_4b88_8378_ee67c0747e52.slice/crio-d3a0900842bb2ad22b935057c96e27fa588e499d1ca673aced3ee5b7a76723ea WatchSource:0}: Error finding container d3a0900842bb2ad22b935057c96e27fa588e499d1ca673aced3ee5b7a76723ea: Status 404 returned error can't find the container with id d3a0900842bb2ad22b935057c96e27fa588e499d1ca673aced3ee5b7a76723ea Aug 05 18:14:16.089940 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:16.089903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" event={"ID":"2fbcf7c5-0700-4b88-8378-ee67c0747e52","Type":"ContainerStarted","Data":"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef"} Aug 05 18:14:16.090377 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:16.089951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" event={"ID":"2fbcf7c5-0700-4b88-8378-ee67c0747e52","Type":"ContainerStarted","Data":"d3a0900842bb2ad22b935057c96e27fa588e499d1ca673aced3ee5b7a76723ea"} Aug 05 18:14:16.108170 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:16.108108 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" podStartSLOduration=1.10808679 podStartE2EDuration="1.10808679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:14:16.108032856 +0000 UTC m=+3112.669318474" watchObservedRunningTime="2026-08-05 18:14:16.10808679 +0000 UTC m=+3112.669372404" Aug 05 18:14:17.715985 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:17.715946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:14:25.399994 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:25.399957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:25.400429 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:25.400010 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:25.412513 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:25.412485 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:26.147716 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:26.147677 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:14:56.930573 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:14:56.930487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:15:09.578289 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:09.578248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:15:21.320917 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:21.320879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:15:25.188387 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:25.188349 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:15:26.189698 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:26.189665 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:15:31.302613 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:31.302578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:15:40.207578 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.207532 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" podUID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" containerName="main" containerID="cri-o://2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0" gracePeriod=45 Aug 05 18:15:40.452657 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.452629 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:15:40.485376 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.485286 2576 generic.go:356] "Generic (PLEG): container finished" podID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" containerID="2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0" exitCode=0 Aug 05 18:15:40.485376 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.485341 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" Aug 05 18:15:40.485376 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.485358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" event={"ID":"6205f1ae-7f25-4f7c-b219-f5db161b83f2","Type":"ContainerDied","Data":"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0"} Aug 05 18:15:40.485579 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.485403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb" event={"ID":"6205f1ae-7f25-4f7c-b219-f5db161b83f2","Type":"ContainerDied","Data":"69c0f95a56b6c1579f114958cc83fcb2a84ba9cafb888f6eca8e5cd16265d825"} Aug 05 18:15:40.485579 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.485427 2576 scope.go:117] "RemoveContainer" containerID="2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0" Aug 05 18:15:40.495112 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.495096 2576 scope.go:117] "RemoveContainer" containerID="2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0" Aug 05 18:15:40.495414 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:15:40.495391 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0\": container with ID starting with 2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0 not found: ID does not exist" containerID="2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0" Aug 05 18:15:40.495504 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.495426 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0"} err="failed to get container status \"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0\": rpc error: code = NotFound desc = could not find container \"2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0\": container with ID starting with 2ec03bb2884b1d35e8f03162bbd4f2ed28c3c655ef7da0957a71142e75bce7a0 not found: ID does not exist" Aug 05 18:15:40.502754 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502737 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.502846 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502770 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.502846 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502794 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.502916 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.502916 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.502997 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.502938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg9vv\" (UniqueName: \"kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv\") pod \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\" (UID: \"6205f1ae-7f25-4f7c-b219-f5db161b83f2\") " Aug 05 18:15:40.503054 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home" (OuterVolumeSpecName: "home") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:40.503116 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503090 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:40.503202 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503181 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:40.503312 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503296 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.503387 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503313 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.503387 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.503349 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.504814 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.504791 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:15:40.504916 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.504894 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm" (OuterVolumeSpecName: "dshm") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:40.504916 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.504905 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv" (OuterVolumeSpecName: "kube-api-access-wg9vv") pod "6205f1ae-7f25-4f7c-b219-f5db161b83f2" (UID: "6205f1ae-7f25-4f7c-b219-f5db161b83f2"). InnerVolumeSpecName "kube-api-access-wg9vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:15:40.604593 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.604560 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9vv\" (UniqueName: \"kubernetes.io/projected/6205f1ae-7f25-4f7c-b219-f5db161b83f2-kube-api-access-wg9vv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.604593 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.604589 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6205f1ae-7f25-4f7c-b219-f5db161b83f2-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.604593 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.604599 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6205f1ae-7f25-4f7c-b219-f5db161b83f2-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:40.750209 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.750135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:15:40.821689 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.821655 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:15:40.825960 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:40.825935 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ba69fc/sub-test-model-ba69fc-kserve-66cbfc755c-r5qfb"] Aug 05 18:15:41.208791 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.208731 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" podUID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" containerName="main" containerID="cri-o://9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef" gracePeriod=45 Aug 05 18:15:41.456722 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.456700 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:15:41.491015 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.490948 2576 generic.go:356] "Generic (PLEG): container finished" podID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" containerID="9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef" exitCode=0 Aug 05 18:15:41.491015 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.491006 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" Aug 05 18:15:41.491185 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.491010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" event={"ID":"2fbcf7c5-0700-4b88-8378-ee67c0747e52","Type":"ContainerDied","Data":"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef"} Aug 05 18:15:41.491185 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.491144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b" event={"ID":"2fbcf7c5-0700-4b88-8378-ee67c0747e52","Type":"ContainerDied","Data":"d3a0900842bb2ad22b935057c96e27fa588e499d1ca673aced3ee5b7a76723ea"} Aug 05 18:15:41.491185 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.491175 2576 scope.go:117] "RemoveContainer" containerID="9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef" Aug 05 18:15:41.499622 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.499602 2576 scope.go:117] "RemoveContainer" containerID="9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef" Aug 05 18:15:41.499886 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:15:41.499868 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef\": container with ID starting with 9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef not found: ID does not exist" containerID="9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef" Aug 05 18:15:41.499955 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.499897 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef"} err="failed to get container status \"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef\": rpc error: code = NotFound desc = could not find container \"9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef\": container with ID starting with 9f8040fa4a6a704958b188f5b82096bca43cd7eed578bf8d95068869f4c595ef not found: ID does not exist" Aug 05 18:15:41.511968 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.511947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512076 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.511979 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512076 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512076 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512058 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512252 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512083 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512252 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512142 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6ddp\" (UniqueName: \"kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp\") pod \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\" (UID: \"2fbcf7c5-0700-4b88-8378-ee67c0747e52\") " Aug 05 18:15:41.512252 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:41.512433 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512357 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:41.512433 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512367 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home" (OuterVolumeSpecName: "home") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:41.512517 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512461 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.512517 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512474 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.512517 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.512483 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.514191 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.514167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm" (OuterVolumeSpecName: "dshm") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:15:41.514191 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.514171 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp" (OuterVolumeSpecName: "kube-api-access-g6ddp") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "kube-api-access-g6ddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:15:41.514379 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.514173 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fbcf7c5-0700-4b88-8378-ee67c0747e52" (UID: "2fbcf7c5-0700-4b88-8378-ee67c0747e52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:15:41.613833 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.613795 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbcf7c5-0700-4b88-8378-ee67c0747e52-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.613833 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.613829 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6ddp\" (UniqueName: \"kubernetes.io/projected/2fbcf7c5-0700-4b88-8378-ee67c0747e52-kube-api-access-g6ddp\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.614030 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.613844 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbcf7c5-0700-4b88-8378-ee67c0747e52-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:15:41.814776 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.814747 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:15:41.819364 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.819339 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/sub-test-model-594b65-kserve-578797979c-9qn9b"] Aug 05 18:15:41.977938 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.977896 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" path="/var/lib/kubelet/pods/2fbcf7c5-0700-4b88-8378-ee67c0747e52/volumes" Aug 05 18:15:41.978413 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:15:41.978392 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" path="/var/lib/kubelet/pods/6205f1ae-7f25-4f7c-b219-f5db161b83f2/volumes" Aug 05 18:16:16.181989 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.181906 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182296 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" containerName="main" Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182309 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" containerName="main" Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182346 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" containerName="main" Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182352 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" containerName="main" Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182405 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fbcf7c5-0700-4b88-8378-ee67c0747e52" containerName="main" Aug 05 18:16:16.182496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.182413 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6205f1ae-7f25-4f7c-b219-f5db161b83f2" containerName="main" Aug 05 18:16:16.185266 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.185250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.187630 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.187607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"default-dockercfg-6dppk\"" Aug 05 18:16:16.187630 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.187628 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"kube-root-ca.crt\"" Aug 05 18:16:16.188533 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.188516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"rate-test-model-594b65-kserve-self-signed-certs\"" Aug 05 18:16:16.188533 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.188523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-594b65\"/\"openshift-service-ca.crt\"" Aug 05 18:16:16.195155 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.195133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:16:16.330485 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.330660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.330660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.330660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.330660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.330821 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.330679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hqp\" (UniqueName: \"kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.431934 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.431905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432126 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.431944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432126 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.431969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432126 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.431993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432126 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.432012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432126 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.432033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hqp\" (UniqueName: \"kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432511 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.432488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432570 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.432508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.432628 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.432569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.434360 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.434340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.434536 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.434522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.439903 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.439885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hqp\" (UniqueName: \"kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp\") pod \"rate-test-model-594b65-kserve-6874667d8c-4zq4j\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.495525 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.495492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:16.623712 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.623687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:16:16.626524 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:16:16.626495 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15180408_8f6e_4560_9551_2d5fd62912dd.slice/crio-bcd9a4604da90ac32b5624f1c24e831d7753124da2bff8f0c4078095755ae273 WatchSource:0}: Error finding container bcd9a4604da90ac32b5624f1c24e831d7753124da2bff8f0c4078095755ae273: Status 404 returned error can't find the container with id bcd9a4604da90ac32b5624f1c24e831d7753124da2bff8f0c4078095755ae273 Aug 05 18:16:16.654290 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:16.654262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" event={"ID":"15180408-8f6e-4560-9551-2d5fd62912dd","Type":"ContainerStarted","Data":"bcd9a4604da90ac32b5624f1c24e831d7753124da2bff8f0c4078095755ae273"} Aug 05 18:16:17.661130 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:17.661093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" event={"ID":"15180408-8f6e-4560-9551-2d5fd62912dd","Type":"ContainerStarted","Data":"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f"} Aug 05 18:16:17.686170 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:17.686114 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" podStartSLOduration=1.686097831 podStartE2EDuration="1.686097831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:16:17.68473685 +0000 UTC m=+3234.246022464" watchObservedRunningTime="2026-08-05 18:16:17.686097831 +0000 UTC m=+3234.247383445" Aug 05 18:16:18.809988 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:18.809951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:16:26.496588 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:26.496546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:26.497091 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:26.496806 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:26.509496 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:26.509469 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:26.715925 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:26.715899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:16:53.804691 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:16:53.804656 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:17:07.015130 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:07.015094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:17:17.316616 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:17.316582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:17:24.123633 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:24.123602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:17:24.136059 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:24.136031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:17:24.381369 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:24.381252 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:17:25.314042 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.314004 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293285424"] Aug 05 18:17:25.317450 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.317430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:25.319780 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.319758 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2xvgc\"" Aug 05 18:17:25.324512 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.324487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293285424"] Aug 05 18:17:25.360374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.360304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qzh\" (UniqueName: \"kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh\") pod \"test-curl-7217-140275293285424\" (UID: \"2bb1df08-6346-4bf2-b50c-c302270678b9\") " pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:25.461154 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.461118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qzh\" (UniqueName: \"kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh\") pod \"test-curl-7217-140275293285424\" (UID: \"2bb1df08-6346-4bf2-b50c-c302270678b9\") " pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:25.470045 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.470014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qzh\" (UniqueName: \"kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh\") pod \"test-curl-7217-140275293285424\" (UID: \"2bb1df08-6346-4bf2-b50c-c302270678b9\") " pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:25.628130 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.628029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:25.763815 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.763791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293285424"] Aug 05 18:17:25.765741 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:25.765711 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb1df08_6346_4bf2_b50c_c302270678b9.slice/crio-9c84c1c2b405114d00a2fdb49777c66e94729801531339dc78a387ac64070f4d WatchSource:0}: Error finding container 9c84c1c2b405114d00a2fdb49777c66e94729801531339dc78a387ac64070f4d: Status 404 returned error can't find the container with id 9c84c1c2b405114d00a2fdb49777c66e94729801531339dc78a387ac64070f4d Aug 05 18:17:25.767024 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.767003 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 18:17:25.982866 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:25.977819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293285424" event={"ID":"2bb1df08-6346-4bf2-b50c-c302270678b9","Type":"ContainerStarted","Data":"9c84c1c2b405114d00a2fdb49777c66e94729801531339dc78a387ac64070f4d"} Aug 05 18:17:26.978619 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:26.978582 2576 generic.go:356] "Generic (PLEG): container finished" podID="2bb1df08-6346-4bf2-b50c-c302270678b9" containerID="f0348888ceee4dd49bdee9bec106a2b2589c0cc6db9ffaaa9acc57cd95f5e44c" exitCode=0 Aug 05 18:17:26.979008 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:26.978645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293285424" event={"ID":"2bb1df08-6346-4bf2-b50c-c302270678b9","Type":"ContainerDied","Data":"f0348888ceee4dd49bdee9bec106a2b2589c0cc6db9ffaaa9acc57cd95f5e44c"} Aug 05 18:17:27.010800 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:27.010767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:17:28.120340 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.120296 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:28.163817 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.163784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293285424_2bb1df08-6346-4bf2-b50c-c302270678b9/test-curl-7217-140275293285424/0.log" Aug 05 18:17:28.186871 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.186839 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5qzh\" (UniqueName: \"kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh\") pod \"2bb1df08-6346-4bf2-b50c-c302270678b9\" (UID: \"2bb1df08-6346-4bf2-b50c-c302270678b9\") " Aug 05 18:17:28.189400 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.189366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh" (OuterVolumeSpecName: "kube-api-access-p5qzh") pod "2bb1df08-6346-4bf2-b50c-c302270678b9" (UID: "2bb1df08-6346-4bf2-b50c-c302270678b9"). InnerVolumeSpecName "kube-api-access-p5qzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:28.192290 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.192263 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293285424"] Aug 05 18:17:28.197883 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.197858 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293285424"] Aug 05 18:17:28.288501 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.288414 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qzh\" (UniqueName: \"kubernetes.io/projected/2bb1df08-6346-4bf2-b50c-c302270678b9-kube-api-access-p5qzh\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:28.316507 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.316470 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293284016"] Aug 05 18:17:28.316867 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.316854 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb1df08-6346-4bf2-b50c-c302270678b9" containerName="test-curl-7217-140275293285424" Aug 05 18:17:28.316913 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.316869 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb1df08-6346-4bf2-b50c-c302270678b9" containerName="test-curl-7217-140275293285424" Aug 05 18:17:28.316948 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.316934 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb1df08-6346-4bf2-b50c-c302270678b9" containerName="test-curl-7217-140275293285424" Aug 05 18:17:28.319832 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.319808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:28.328859 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.328830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293284016"] Aug 05 18:17:28.389844 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.389798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7px7\" (UniqueName: \"kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7\") pod \"test-curl-7217-140275293284016\" (UID: \"2521d4d8-4a49-4718-8217-d3e957db83f7\") " pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:28.490655 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.490620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7px7\" (UniqueName: \"kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7\") pod \"test-curl-7217-140275293284016\" (UID: \"2521d4d8-4a49-4718-8217-d3e957db83f7\") " pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:28.500288 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.500254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7px7\" (UniqueName: \"kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7\") pod \"test-curl-7217-140275293284016\" (UID: \"2521d4d8-4a49-4718-8217-d3e957db83f7\") " pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:28.629828 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.629733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:28.771632 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.771609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293284016"] Aug 05 18:17:28.774171 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:28.774143 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2521d4d8_4a49_4718_8217_d3e957db83f7.slice/crio-c0eec0a661c604ac982b1332dd0db0988d99682cee7bb7934231bf98bfae1261 WatchSource:0}: Error finding container c0eec0a661c604ac982b1332dd0db0988d99682cee7bb7934231bf98bfae1261: Status 404 returned error can't find the container with id c0eec0a661c604ac982b1332dd0db0988d99682cee7bb7934231bf98bfae1261 Aug 05 18:17:28.989808 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.989766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293284016" event={"ID":"2521d4d8-4a49-4718-8217-d3e957db83f7","Type":"ContainerStarted","Data":"c0eec0a661c604ac982b1332dd0db0988d99682cee7bb7934231bf98bfae1261"} Aug 05 18:17:28.991029 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.991007 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c84c1c2b405114d00a2fdb49777c66e94729801531339dc78a387ac64070f4d" Aug 05 18:17:28.991153 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:28.991044 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:29.002126 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:17:29.002096 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7217-140275293285424\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="2bb1df08-6346-4bf2-b50c-c302270678b9" pod="opendatahub/test-curl-7217-140275293285424" Aug 05 18:17:29.978923 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:29.978892 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb1df08-6346-4bf2-b50c-c302270678b9" path="/var/lib/kubelet/pods/2bb1df08-6346-4bf2-b50c-c302270678b9/volumes" Aug 05 18:17:29.997423 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:29.997386 2576 generic.go:356] "Generic (PLEG): container finished" podID="2521d4d8-4a49-4718-8217-d3e957db83f7" containerID="9aca38b3870f511ba31911135d550e8897280e489b669c0f86ace07ddc06ddec" exitCode=0 Aug 05 18:17:29.997585 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:29.997478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293284016" event={"ID":"2521d4d8-4a49-4718-8217-d3e957db83f7","Type":"ContainerDied","Data":"9aca38b3870f511ba31911135d550e8897280e489b669c0f86ace07ddc06ddec"} Aug 05 18:17:31.138646 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.138619 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:31.159092 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.159050 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293284016_2521d4d8-4a49-4718-8217-d3e957db83f7/test-curl-7217-140275293284016/0.log" Aug 05 18:17:31.189380 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.184068 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293284016"] Aug 05 18:17:31.191940 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.191907 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293284016"] Aug 05 18:17:31.217293 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.217263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7px7\" (UniqueName: \"kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7\") pod \"2521d4d8-4a49-4718-8217-d3e957db83f7\" (UID: \"2521d4d8-4a49-4718-8217-d3e957db83f7\") " Aug 05 18:17:31.219499 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.219469 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7" (OuterVolumeSpecName: "kube-api-access-g7px7") pod "2521d4d8-4a49-4718-8217-d3e957db83f7" (UID: "2521d4d8-4a49-4718-8217-d3e957db83f7"). InnerVolumeSpecName "kube-api-access-g7px7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:31.318087 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.317996 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7px7\" (UniqueName: \"kubernetes.io/projected/2521d4d8-4a49-4718-8217-d3e957db83f7-kube-api-access-g7px7\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.337238 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.337201 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:31.337594 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.337580 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2521d4d8-4a49-4718-8217-d3e957db83f7" containerName="test-curl-7217-140275293284016" Aug 05 18:17:31.337656 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.337595 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2521d4d8-4a49-4718-8217-d3e957db83f7" containerName="test-curl-7217-140275293284016" Aug 05 18:17:31.337689 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.337674 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2521d4d8-4a49-4718-8217-d3e957db83f7" containerName="test-curl-7217-140275293284016" Aug 05 18:17:31.340621 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.340606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:31.347272 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.347243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:31.419480 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.419429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv6z\" (UniqueName: \"kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z\") pod \"test-curl-7217-140275293355296\" (UID: \"4a8857bd-444f-4575-9938-fd3030eb6e58\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:31.520757 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.520723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv6z\" (UniqueName: \"kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z\") pod \"test-curl-7217-140275293355296\" (UID: \"4a8857bd-444f-4575-9938-fd3030eb6e58\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:31.529364 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.529336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv6z\" (UniqueName: \"kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z\") pod \"test-curl-7217-140275293355296\" (UID: \"4a8857bd-444f-4575-9938-fd3030eb6e58\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:31.651546 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.651450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:31.781683 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.781606 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:31.783920 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:31.783893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a8857bd_444f_4575_9938_fd3030eb6e58.slice/crio-42d85a2b3cc954f8c9c9be81154b0d044e045465a3f98636abe2968b7f408ef5 WatchSource:0}: Error finding container 42d85a2b3cc954f8c9c9be81154b0d044e045465a3f98636abe2968b7f408ef5: Status 404 returned error can't find the container with id 42d85a2b3cc954f8c9c9be81154b0d044e045465a3f98636abe2968b7f408ef5 Aug 05 18:17:31.977821 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:31.977786 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2521d4d8-4a49-4718-8217-d3e957db83f7" path="/var/lib/kubelet/pods/2521d4d8-4a49-4718-8217-d3e957db83f7/volumes" Aug 05 18:17:32.007940 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:32.007904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293355296" event={"ID":"4a8857bd-444f-4575-9938-fd3030eb6e58","Type":"ContainerStarted","Data":"42d85a2b3cc954f8c9c9be81154b0d044e045465a3f98636abe2968b7f408ef5"} Aug 05 18:17:32.009160 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:32.009139 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293284016" Aug 05 18:17:32.009281 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:32.009164 2576 scope.go:117] "RemoveContainer" containerID="9aca38b3870f511ba31911135d550e8897280e489b669c0f86ace07ddc06ddec" Aug 05 18:17:33.015494 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:33.015462 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a8857bd-444f-4575-9938-fd3030eb6e58" containerID="31ca80e7cb04e5db945360dad49c37b0facf747da4af2d0bf9a873590f229d93" exitCode=0 Aug 05 18:17:33.016004 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:33.015541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293355296" event={"ID":"4a8857bd-444f-4575-9938-fd3030eb6e58","Type":"ContainerDied","Data":"31ca80e7cb04e5db945360dad49c37b0facf747da4af2d0bf9a873590f229d93"} Aug 05 18:17:34.157717 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.157692 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.179058 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.179030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293355296_4a8857bd-444f-4575-9938-fd3030eb6e58/test-curl-7217-140275293355296/0.log" Aug 05 18:17:34.200551 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.200508 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:34.203876 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.203849 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:34.245774 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.245742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdv6z\" (UniqueName: \"kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z\") pod \"4a8857bd-444f-4575-9938-fd3030eb6e58\" (UID: \"4a8857bd-444f-4575-9938-fd3030eb6e58\") " Aug 05 18:17:34.247791 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.247762 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z" (OuterVolumeSpecName: "kube-api-access-bdv6z") pod "4a8857bd-444f-4575-9938-fd3030eb6e58" (UID: "4a8857bd-444f-4575-9938-fd3030eb6e58"). InnerVolumeSpecName "kube-api-access-bdv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:34.317550 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.317466 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:34.317837 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.317823 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8857bd-444f-4575-9938-fd3030eb6e58" containerName="test-curl-7217-140275293355296" Aug 05 18:17:34.317885 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.317838 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8857bd-444f-4575-9938-fd3030eb6e58" containerName="test-curl-7217-140275293355296" Aug 05 18:17:34.317920 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.317912 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8857bd-444f-4575-9938-fd3030eb6e58" containerName="test-curl-7217-140275293355296" Aug 05 18:17:34.320774 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.320757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.325703 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.325673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:34.347395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.347359 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdv6z\" (UniqueName: \"kubernetes.io/projected/4a8857bd-444f-4575-9938-fd3030eb6e58-kube-api-access-bdv6z\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:34.448187 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.448151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjpd\" (UniqueName: \"kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd\") pod \"test-curl-7217-140275293355296\" (UID: \"7e0b4521-3f58-4849-be93-0ec278e55150\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.548787 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.548744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjpd\" (UniqueName: \"kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd\") pod \"test-curl-7217-140275293355296\" (UID: \"7e0b4521-3f58-4849-be93-0ec278e55150\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.558056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.558026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjpd\" (UniqueName: \"kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd\") pod \"test-curl-7217-140275293355296\" (UID: \"7e0b4521-3f58-4849-be93-0ec278e55150\") " pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.631709 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.631617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:34.772583 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:34.772440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:34.774900 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:34.774870 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e0b4521_3f58_4849_be93_0ec278e55150.slice/crio-4b73d3c84ef9f81d192594a11a7b374b25469e4d6cd155286cd375df1eabadf5 WatchSource:0}: Error finding container 4b73d3c84ef9f81d192594a11a7b374b25469e4d6cd155286cd375df1eabadf5: Status 404 returned error can't find the container with id 4b73d3c84ef9f81d192594a11a7b374b25469e4d6cd155286cd375df1eabadf5 Aug 05 18:17:35.027667 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:35.027623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293355296" event={"ID":"7e0b4521-3f58-4849-be93-0ec278e55150","Type":"ContainerStarted","Data":"4b73d3c84ef9f81d192594a11a7b374b25469e4d6cd155286cd375df1eabadf5"} Aug 05 18:17:35.028858 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:35.028834 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d85a2b3cc954f8c9c9be81154b0d044e045465a3f98636abe2968b7f408ef5" Aug 05 18:17:35.028858 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:35.028853 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:35.040707 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:35.040675 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7217-140275293355296" oldPodUID="4a8857bd-444f-4575-9938-fd3030eb6e58" podUID="7e0b4521-3f58-4849-be93-0ec278e55150" Aug 05 18:17:35.978179 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:35.978141 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8857bd-444f-4575-9938-fd3030eb6e58" path="/var/lib/kubelet/pods/4a8857bd-444f-4575-9938-fd3030eb6e58/volumes" Aug 05 18:17:36.035035 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:36.035000 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e0b4521-3f58-4849-be93-0ec278e55150" containerID="480908c90e4dd84bee2ff35a439b89f21e84c66a8533c05532fad26a043add6c" exitCode=0 Aug 05 18:17:36.035214 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:36.035087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293355296" event={"ID":"7e0b4521-3f58-4849-be93-0ec278e55150","Type":"ContainerDied","Data":"480908c90e4dd84bee2ff35a439b89f21e84c66a8533c05532fad26a043add6c"} Aug 05 18:17:37.173787 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.173760 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:37.194898 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.194866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293355296_7e0b4521-3f58-4849-be93-0ec278e55150/test-curl-7217-140275293355296/0.log" Aug 05 18:17:37.222394 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.222352 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:37.227720 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.227692 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293355296"] Aug 05 18:17:37.277264 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.277235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxjpd\" (UniqueName: \"kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd\") pod \"7e0b4521-3f58-4849-be93-0ec278e55150\" (UID: \"7e0b4521-3f58-4849-be93-0ec278e55150\") " Aug 05 18:17:37.279395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.279369 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd" (OuterVolumeSpecName: "kube-api-access-gxjpd") pod "7e0b4521-3f58-4849-be93-0ec278e55150" (UID: "7e0b4521-3f58-4849-be93-0ec278e55150"). InnerVolumeSpecName "kube-api-access-gxjpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:37.344285 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.344199 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:37.344616 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.344601 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0b4521-3f58-4849-be93-0ec278e55150" containerName="test-curl-7217-140275293355296" Aug 05 18:17:37.344668 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.344617 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0b4521-3f58-4849-be93-0ec278e55150" containerName="test-curl-7217-140275293355296" Aug 05 18:17:37.344701 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.344678 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0b4521-3f58-4849-be93-0ec278e55150" containerName="test-curl-7217-140275293355296" Aug 05 18:17:37.347666 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.347649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:37.352311 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.352282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:37.378301 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.378268 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjpd\" (UniqueName: \"kubernetes.io/projected/7e0b4521-3f58-4849-be93-0ec278e55150-kube-api-access-gxjpd\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:37.479149 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.479114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcc4v\" (UniqueName: \"kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v\") pod \"test-curl-7217-140275293507920\" (UID: \"5b12a1f4-7b17-417c-91c6-2e19d3242973\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:37.580518 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.580482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcc4v\" (UniqueName: \"kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v\") pod \"test-curl-7217-140275293507920\" (UID: \"5b12a1f4-7b17-417c-91c6-2e19d3242973\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:37.589054 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.589020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcc4v\" (UniqueName: \"kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v\") pod \"test-curl-7217-140275293507920\" (UID: \"5b12a1f4-7b17-417c-91c6-2e19d3242973\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:37.658522 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.658436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:37.791760 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.791731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:37.793057 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:37.793026 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b12a1f4_7b17_417c_91c6_2e19d3242973.slice/crio-41b708fc177ee3dbae6c4a6744e6f28ddb63339efc207d506c7cb383ef1bfbc4 WatchSource:0}: Error finding container 41b708fc177ee3dbae6c4a6744e6f28ddb63339efc207d506c7cb383ef1bfbc4: Status 404 returned error can't find the container with id 41b708fc177ee3dbae6c4a6744e6f28ddb63339efc207d506c7cb383ef1bfbc4 Aug 05 18:17:37.978375 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:37.978349 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0b4521-3f58-4849-be93-0ec278e55150" path="/var/lib/kubelet/pods/7e0b4521-3f58-4849-be93-0ec278e55150/volumes" Aug 05 18:17:38.045926 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:38.045892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293507920" event={"ID":"5b12a1f4-7b17-417c-91c6-2e19d3242973","Type":"ContainerStarted","Data":"41b708fc177ee3dbae6c4a6744e6f28ddb63339efc207d506c7cb383ef1bfbc4"} Aug 05 18:17:38.047090 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:38.047068 2576 scope.go:117] "RemoveContainer" containerID="480908c90e4dd84bee2ff35a439b89f21e84c66a8533c05532fad26a043add6c" Aug 05 18:17:38.047090 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:38.047083 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293355296" Aug 05 18:17:39.057674 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.057640 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b12a1f4-7b17-417c-91c6-2e19d3242973" containerID="af62c0ea58b18b96fa5a3fa685da52ed79b009e2e669ea999e703860c4be5fc8" exitCode=0 Aug 05 18:17:39.058119 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.057722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293507920" event={"ID":"5b12a1f4-7b17-417c-91c6-2e19d3242973","Type":"ContainerDied","Data":"af62c0ea58b18b96fa5a3fa685da52ed79b009e2e669ea999e703860c4be5fc8"} Aug 05 18:17:39.400563 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.400477 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" podUID="15180408-8f6e-4560-9551-2d5fd62912dd" containerName="main" containerID="cri-o://46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f" gracePeriod=45 Aug 05 18:17:39.647473 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.647444 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:17:39.802615 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802580 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.802615 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802618 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5hqp\" (UniqueName: \"kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.802892 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.802892 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802674 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.802892 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802715 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.802892 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802766 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home\") pod \"15180408-8f6e-4560-9551-2d5fd62912dd\" (UID: \"15180408-8f6e-4560-9551-2d5fd62912dd\") " Aug 05 18:17:39.803103 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802937 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:39.803103 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.802951 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:39.803194 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.803099 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home" (OuterVolumeSpecName: "home") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:39.803705 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.803681 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:39.803826 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.803709 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:39.805352 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.805299 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp" (OuterVolumeSpecName: "kube-api-access-l5hqp") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "kube-api-access-l5hqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:39.811470 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.811440 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm" (OuterVolumeSpecName: "dshm") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:39.812199 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.812172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15180408-8f6e-4560-9551-2d5fd62912dd" (UID: "15180408-8f6e-4560-9551-2d5fd62912dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:17:39.904604 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.904569 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hqp\" (UniqueName: \"kubernetes.io/projected/15180408-8f6e-4560-9551-2d5fd62912dd-kube-api-access-l5hqp\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:39.904604 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.904602 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15180408-8f6e-4560-9551-2d5fd62912dd-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:39.904814 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.904615 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:39.904814 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:39.904623 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15180408-8f6e-4560-9551-2d5fd62912dd-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:40.064922 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.064818 2576 generic.go:356] "Generic (PLEG): container finished" podID="15180408-8f6e-4560-9551-2d5fd62912dd" containerID="46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f" exitCode=0 Aug 05 18:17:40.064922 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.064880 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" Aug 05 18:17:40.065492 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.064918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" event={"ID":"15180408-8f6e-4560-9551-2d5fd62912dd","Type":"ContainerDied","Data":"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f"} Aug 05 18:17:40.065492 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.064957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j" event={"ID":"15180408-8f6e-4560-9551-2d5fd62912dd","Type":"ContainerDied","Data":"bcd9a4604da90ac32b5624f1c24e831d7753124da2bff8f0c4078095755ae273"} Aug 05 18:17:40.065492 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.064975 2576 scope.go:117] "RemoveContainer" containerID="46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f" Aug 05 18:17:40.074541 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.074511 2576 scope.go:117] "RemoveContainer" containerID="46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f" Aug 05 18:17:40.074887 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:17:40.074861 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f\": container with ID starting with 46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f not found: ID does not exist" containerID="46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f" Aug 05 18:17:40.075004 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.074895 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f"} err="failed to get container status \"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f\": rpc error: code = NotFound desc = could not find container \"46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f\": container with ID starting with 46901c35463bd227dcbeacc59fc0d3fb633c424f9b2c835be2fd6d25097b7a1f not found: ID does not exist" Aug 05 18:17:40.081950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.081922 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:17:40.085583 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.085560 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-594b65/rate-test-model-594b65-kserve-6874667d8c-4zq4j"] Aug 05 18:17:40.192012 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.191980 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.213486 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.213458 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293507920_5b12a1f4-7b17-417c-91c6-2e19d3242973/test-curl-7217-140275293507920/0.log" Aug 05 18:17:40.235499 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.235467 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:40.238942 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.238917 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:40.308497 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.308460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcc4v\" (UniqueName: \"kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v\") pod \"5b12a1f4-7b17-417c-91c6-2e19d3242973\" (UID: \"5b12a1f4-7b17-417c-91c6-2e19d3242973\") " Aug 05 18:17:40.310556 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.310527 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v" (OuterVolumeSpecName: "kube-api-access-kcc4v") pod "5b12a1f4-7b17-417c-91c6-2e19d3242973" (UID: "5b12a1f4-7b17-417c-91c6-2e19d3242973"). InnerVolumeSpecName "kube-api-access-kcc4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:40.358559 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358462 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:40.358850 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358836 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15180408-8f6e-4560-9551-2d5fd62912dd" containerName="main" Aug 05 18:17:40.358901 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358852 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15180408-8f6e-4560-9551-2d5fd62912dd" containerName="main" Aug 05 18:17:40.358901 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358869 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b12a1f4-7b17-417c-91c6-2e19d3242973" containerName="test-curl-7217-140275293507920" Aug 05 18:17:40.358901 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358876 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b12a1f4-7b17-417c-91c6-2e19d3242973" containerName="test-curl-7217-140275293507920" Aug 05 18:17:40.359009 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358950 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b12a1f4-7b17-417c-91c6-2e19d3242973" containerName="test-curl-7217-140275293507920" Aug 05 18:17:40.359009 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.358960 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15180408-8f6e-4560-9551-2d5fd62912dd" containerName="main" Aug 05 18:17:40.361946 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.361930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.369108 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.369073 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:40.409893 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.409857 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcc4v\" (UniqueName: \"kubernetes.io/projected/5b12a1f4-7b17-417c-91c6-2e19d3242973-kube-api-access-kcc4v\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:40.511169 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.511129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5n7b\" (UniqueName: \"kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b\") pod \"test-curl-7217-140275293507920\" (UID: \"715de4fc-02d6-41f5-8175-aaf502db0633\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.612724 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.612626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5n7b\" (UniqueName: \"kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b\") pod \"test-curl-7217-140275293507920\" (UID: \"715de4fc-02d6-41f5-8175-aaf502db0633\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.621473 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.621447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5n7b\" (UniqueName: \"kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b\") pod \"test-curl-7217-140275293507920\" (UID: \"715de4fc-02d6-41f5-8175-aaf502db0633\") " pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.671895 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.671851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:40.802208 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:40.802177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:40.804585 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:40.804557 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715de4fc_02d6_41f5_8175_aaf502db0633.slice/crio-1316c49a0b26e6f49cf2f23bd7f02bd08f7ebb3d4ebae2d9ac00f499b63511b7 WatchSource:0}: Error finding container 1316c49a0b26e6f49cf2f23bd7f02bd08f7ebb3d4ebae2d9ac00f499b63511b7: Status 404 returned error can't find the container with id 1316c49a0b26e6f49cf2f23bd7f02bd08f7ebb3d4ebae2d9ac00f499b63511b7 Aug 05 18:17:41.071514 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.071487 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:41.071514 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.071507 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b708fc177ee3dbae6c4a6744e6f28ddb63339efc207d506c7cb383ef1bfbc4" Aug 05 18:17:41.073492 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.073467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293507920" event={"ID":"715de4fc-02d6-41f5-8175-aaf502db0633","Type":"ContainerStarted","Data":"1316c49a0b26e6f49cf2f23bd7f02bd08f7ebb3d4ebae2d9ac00f499b63511b7"} Aug 05 18:17:41.083528 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.083502 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7217-140275293507920" oldPodUID="5b12a1f4-7b17-417c-91c6-2e19d3242973" podUID="715de4fc-02d6-41f5-8175-aaf502db0633" Aug 05 18:17:41.977919 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.977876 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15180408-8f6e-4560-9551-2d5fd62912dd" path="/var/lib/kubelet/pods/15180408-8f6e-4560-9551-2d5fd62912dd/volumes" Aug 05 18:17:41.978403 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:41.978384 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b12a1f4-7b17-417c-91c6-2e19d3242973" path="/var/lib/kubelet/pods/5b12a1f4-7b17-417c-91c6-2e19d3242973/volumes" Aug 05 18:17:42.080868 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:42.080831 2576 generic.go:356] "Generic (PLEG): container finished" podID="715de4fc-02d6-41f5-8175-aaf502db0633" containerID="b23bf67d4fb6a2737adae0ee34e4055f4b03d2c716ebbd71e6347dcbb8d32cc5" exitCode=0 Aug 05 18:17:42.081303 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:42.080911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7217-140275293507920" event={"ID":"715de4fc-02d6-41f5-8175-aaf502db0633","Type":"ContainerDied","Data":"b23bf67d4fb6a2737adae0ee34e4055f4b03d2c716ebbd71e6347dcbb8d32cc5"} Aug 05 18:17:43.226337 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.226303 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:43.247148 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.247122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7217-140275293507920_715de4fc-02d6-41f5-8175-aaf502db0633/test-curl-7217-140275293507920/0.log" Aug 05 18:17:43.268978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.268943 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:43.275805 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.275779 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7217-140275293507920"] Aug 05 18:17:43.339026 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.338942 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5n7b\" (UniqueName: \"kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b\") pod \"715de4fc-02d6-41f5-8175-aaf502db0633\" (UID: \"715de4fc-02d6-41f5-8175-aaf502db0633\") " Aug 05 18:17:43.340975 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.340949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b" (OuterVolumeSpecName: "kube-api-access-c5n7b") pod "715de4fc-02d6-41f5-8175-aaf502db0633" (UID: "715de4fc-02d6-41f5-8175-aaf502db0633"). InnerVolumeSpecName "kube-api-access-c5n7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:43.439845 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.439800 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5n7b\" (UniqueName: \"kubernetes.io/projected/715de4fc-02d6-41f5-8175-aaf502db0633-kube-api-access-c5n7b\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:17:43.979395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:43.979360 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715de4fc-02d6-41f5-8175-aaf502db0633" path="/var/lib/kubelet/pods/715de4fc-02d6-41f5-8175-aaf502db0633/volumes" Aug 05 18:17:44.091851 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:44.091819 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7217-140275293507920" Aug 05 18:17:44.092024 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:44.091828 2576 scope.go:117] "RemoveContainer" containerID="b23bf67d4fb6a2737adae0ee34e4055f4b03d2c716ebbd71e6347dcbb8d32cc5" Aug 05 18:17:48.470176 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.470077 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:17:48.470661 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.470638 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715de4fc-02d6-41f5-8175-aaf502db0633" containerName="test-curl-7217-140275293507920" Aug 05 18:17:48.470737 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.470666 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="715de4fc-02d6-41f5-8175-aaf502db0633" containerName="test-curl-7217-140275293507920" Aug 05 18:17:48.470792 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.470781 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="715de4fc-02d6-41f5-8175-aaf502db0633" containerName="test-curl-7217-140275293507920" Aug 05 18:17:48.475151 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.475128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.477841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.477820 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6427fe\"" Aug 05 18:17:48.487920 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.487893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:17:48.586769 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.586732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srqv\" (UniqueName: \"kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.586769 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.586774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.687755 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.687717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srqv\" (UniqueName: \"kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.687755 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.687758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.688547 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.688525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.697866 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.697834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srqv\" (UniqueName: \"kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv\") pod \"payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.785628 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.785524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:48.927767 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:48.927586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:17:48.930868 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:17:48.930841 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac873ca_d817_4f60_8d92_a2ac5bd683ea.slice/crio-fcf642a6eecb0d7ed5b19568c0f78d15b75f005f81f62849c0f4d2dc5f8ecde4 WatchSource:0}: Error finding container fcf642a6eecb0d7ed5b19568c0f78d15b75f005f81f62849c0f4d2dc5f8ecde4: Status 404 returned error can't find the container with id fcf642a6eecb0d7ed5b19568c0f78d15b75f005f81f62849c0f4d2dc5f8ecde4 Aug 05 18:17:49.120256 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:49.120165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" event={"ID":"4ac873ca-d817-4f60-8d92-a2ac5bd683ea","Type":"ContainerStarted","Data":"fcf642a6eecb0d7ed5b19568c0f78d15b75f005f81f62849c0f4d2dc5f8ecde4"} Aug 05 18:17:51.131395 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:51.131363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" event={"ID":"4ac873ca-d817-4f60-8d92-a2ac5bd683ea","Type":"ContainerStarted","Data":"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060"} Aug 05 18:17:51.131821 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:51.131437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:17:51.151562 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:51.151516 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" podStartSLOduration=1.493135627 podStartE2EDuration="3.151501109s" totalImagesPullingTime="1.658365482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:17:51.148929897 +0000 UTC m=+3327.710215512" watchObservedRunningTime="2026-08-05 18:17:51.151501109 +0000 UTC m=+3327.712786722" Aug 05 18:17:51.499466 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:17:51.499426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:18:02.139180 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:02.139151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:18:16.966783 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.966735 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:18:16.970953 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.970924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:16.973381 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.973342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:18:16.973530 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.973416 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a3125a-serving-cert\"" Aug 05 18:18:16.973530 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.973462 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9f8vc\"" Aug 05 18:18:16.974137 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.974121 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 18:18:16.981904 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:16.981879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:18:17.046572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.046538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.046572 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.046579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.046805 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.046652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2w6\" (UniqueName: \"kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.147716 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.147682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.147716 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.147720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.147944 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.147844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2w6\" (UniqueName: \"kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.148359 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.148314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.150070 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.150049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.156989 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.156967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2w6\" (UniqueName: \"kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6\") pod \"maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.282792 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.282699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:17.426623 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.426588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:18:17.430060 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:18:17.430015 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4b4db0_826c_43bf_ac08_809de254ebe9.slice/crio-dfe092dd75022350cf720bcaf968978ad4db4e148d0cc5aecd8282eabaa2b1ed WatchSource:0}: Error finding container dfe092dd75022350cf720bcaf968978ad4db4e148d0cc5aecd8282eabaa2b1ed: Status 404 returned error can't find the container with id dfe092dd75022350cf720bcaf968978ad4db4e148d0cc5aecd8282eabaa2b1ed Aug 05 18:18:17.769471 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.769438 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:18:17.774405 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.774381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.777080 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.777054 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a3125a\"" Aug 05 18:18:17.777236 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.777219 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a3125a-dockercfg-k96z9\"" Aug 05 18:18:17.789332 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.789292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:18:17.855779 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.855743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.855966 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.855797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gs2c\" (UniqueName: \"kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.868738 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.868704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:18:17.879421 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.879390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:17.900703 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.900669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:18:17.957203 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.957163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:17.957404 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.957292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.957404 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.957383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gs2c\" (UniqueName: \"kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.957529 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.957404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7h5j\" (UniqueName: \"kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:17.958122 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.958100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:17.986375 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:17.986347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gs2c\" (UniqueName: \"kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c\") pod \"payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:18.058425 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.058314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7h5j\" (UniqueName: \"kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:18.058608 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.058446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:18.059207 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.059183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:18.072067 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.072036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7h5j\" (UniqueName: \"kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j\") pod \"payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:18.085400 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.085369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:18.189704 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.189664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:18.259898 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.259619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:18:18.271784 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.271718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" event={"ID":"db4b4db0-826c-43bf-ac08-809de254ebe9","Type":"ContainerStarted","Data":"dfe092dd75022350cf720bcaf968978ad4db4e148d0cc5aecd8282eabaa2b1ed"} Aug 05 18:18:18.399300 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:18.399214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:18:18.402644 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:18:18.402613 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281cc71c_512b_44e5_9a7d_8c734ad74909.slice/crio-2565890e80b134d8af45d77493e91d0dc34d0f53bf18460798bf5c402f334449 WatchSource:0}: Error finding container 2565890e80b134d8af45d77493e91d0dc34d0f53bf18460798bf5c402f334449: Status 404 returned error can't find the container with id 2565890e80b134d8af45d77493e91d0dc34d0f53bf18460798bf5c402f334449 Aug 05 18:18:19.280234 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.280174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" event={"ID":"281cc71c-512b-44e5-9a7d-8c734ad74909","Type":"ContainerStarted","Data":"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07"} Aug 05 18:18:19.280722 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.280264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" event={"ID":"281cc71c-512b-44e5-9a7d-8c734ad74909","Type":"ContainerStarted","Data":"2565890e80b134d8af45d77493e91d0dc34d0f53bf18460798bf5c402f334449"} Aug 05 18:18:19.280722 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.280292 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:19.282073 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.282045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" event={"ID":"e4b4df30-7995-4387-8a0a-e6749b3dc97f","Type":"ContainerStarted","Data":"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd"} Aug 05 18:18:19.282189 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.282097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" event={"ID":"e4b4df30-7995-4387-8a0a-e6749b3dc97f","Type":"ContainerStarted","Data":"f714287f77a5658b1d9a9e2aa7f4ed45b020d8c93c79ff8ea2a614b8c766b0b2"} Aug 05 18:18:19.282189 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.282161 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:19.298289 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.298237 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" podStartSLOduration=2.29821942 podStartE2EDuration="2.29821942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:18:19.296502658 +0000 UTC m=+3355.857788271" watchObservedRunningTime="2026-08-05 18:18:19.29821942 +0000 UTC m=+3355.859505012" Aug 05 18:18:19.314032 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:19.313979 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" podStartSLOduration=2.31395983 podStartE2EDuration="2.31395983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:18:19.311774969 +0000 UTC m=+3355.873060583" watchObservedRunningTime="2026-08-05 18:18:19.31395983 +0000 UTC m=+3355.875245443" Aug 05 18:18:20.113680 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:20.113595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:18:20.288729 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:20.288689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" event={"ID":"db4b4db0-826c-43bf-ac08-809de254ebe9","Type":"ContainerStarted","Data":"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095"} Aug 05 18:18:20.315410 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:20.315351 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" podStartSLOduration=1.997817897 podStartE2EDuration="4.315315307s" totalImagesPullingTime="2.31749741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:18:20.314193745 +0000 UTC m=+3356.875479359" watchObservedRunningTime="2026-08-05 18:18:20.315315307 +0000 UTC m=+3356.876600915" Aug 05 18:18:21.293680 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:21.293653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:27.304759 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:27.304724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:18:30.290703 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:30.290669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:18:30.291090 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:30.290793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:18:47.594860 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:18:47.594787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm_281cc71c-512b-44e5-9a7d-8c734ad74909/payload-processing/0.log" Aug 05 18:19:10.779170 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.779137 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:19:10.782885 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.782865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.786270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.786249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6427fe\"/\"kube-root-ca.crt\"" Aug 05 18:19:10.786270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.786264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6427fe\"/\"ipp-route-6427fe-kserve-self-signed-certs\"" Aug 05 18:19:10.786485 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.786264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6427fe\"/\"default-dockercfg-88lbd\"" Aug 05 18:19:10.786485 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.786312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6427fe\"/\"openshift-service-ca.crt\"" Aug 05 18:19:10.792927 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.792903 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:19:10.952855 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.952812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2t4\" (UniqueName: \"kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.953042 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.952871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.953042 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.952939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.953042 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.953013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.953168 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.953089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:10.953168 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:10.953117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053767 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2t4\" (UniqueName: \"kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.053950 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.053932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.054294 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.054240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.054294 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.054288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.054436 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.054362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.055984 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.055963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.056396 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.056379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.062066 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.062044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2t4\" (UniqueName: \"kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4\") pod \"ipp-route-6427fe-kserve-bf8977d4c-n7wdk\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.092799 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.092769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:11.230744 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.230710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:19:11.234965 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:19:11.234932 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5859b046_9924_4df1_8269_d2c2e87db3d2.slice/crio-592304fb31f8c858f16dc0807998d89db3cc85f3617a527e4bb21902e0fcfc65 WatchSource:0}: Error finding container 592304fb31f8c858f16dc0807998d89db3cc85f3617a527e4bb21902e0fcfc65: Status 404 returned error can't find the container with id 592304fb31f8c858f16dc0807998d89db3cc85f3617a527e4bb21902e0fcfc65 Aug 05 18:19:11.537542 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.537507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" event={"ID":"5859b046-9924-4df1-8269-d2c2e87db3d2","Type":"ContainerStarted","Data":"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced"} Aug 05 18:19:11.537542 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.537551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" event={"ID":"5859b046-9924-4df1-8269-d2c2e87db3d2","Type":"ContainerStarted","Data":"592304fb31f8c858f16dc0807998d89db3cc85f3617a527e4bb21902e0fcfc65"} Aug 05 18:19:11.557519 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:11.557462 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" podStartSLOduration=1.557445293 podStartE2EDuration="1.557445293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:19:11.555590529 +0000 UTC m=+3408.116876144" watchObservedRunningTime="2026-08-05 18:19:11.557445293 +0000 UTC m=+3408.118730969" Aug 05 18:19:14.025780 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:14.025741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:19:21.093295 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:21.093206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:21.093295 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:21.093264 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:21.105869 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:21.105835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:21.603466 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:21.603439 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:19:51.922745 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:19:51.922706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:20:03.001210 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:03.001176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc_4ac873ca-d817-4f60-8d92-a2ac5bd683ea/payload-processing/0.log" Aug 05 18:20:03.244490 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:03.244459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm_281cc71c-512b-44e5-9a7d-8c734ad74909/payload-processing/0.log" Aug 05 18:20:08.353250 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.353214 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:08.356887 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.356862 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:08.357038 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.356989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.360339 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.360303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.360482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.360356 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2696ea\"" Aug 05 18:20:08.360482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.360403 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2696ea-dockercfg-s42sv\"" Aug 05 18:20:08.373283 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.373256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:08.384369 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.384314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.384524 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.384434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.384595 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.384549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txms8\" (UniqueName: \"kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.384595 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.384586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4sk\" (UniqueName: \"kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.387199 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.387171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:08.485687 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.485651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4sk\" (UniqueName: \"kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.485876 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.485730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.485876 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.485767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.485876 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.485847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txms8\" (UniqueName: \"kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.486631 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.486604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.486749 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.486603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.501928 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.501901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4sk\" (UniqueName: \"kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk\") pod \"payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.502143 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.502120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txms8\" (UniqueName: \"kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8\") pod \"payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.668400 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.668275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:08.675692 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.675668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:08.832575 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.832549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:08.836023 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:20:08.835993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fca617_a61f_4da2_9511_6e5e735bb56b.slice/crio-80ce9f849175c639d27ea3933dda49fe98c74bb9e2457ee750c77b663dc4cba6 WatchSource:0}: Error finding container 80ce9f849175c639d27ea3933dda49fe98c74bb9e2457ee750c77b663dc4cba6: Status 404 returned error can't find the container with id 80ce9f849175c639d27ea3933dda49fe98c74bb9e2457ee750c77b663dc4cba6 Aug 05 18:20:08.849009 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:08.848951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:08.851832 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:20:08.851800 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01501edf_da8a_4d1a_9f55_90543201c206.slice/crio-306871116c73fa530ec9b78a923c016e2124ec686a28f07779d4ea9850098a86 WatchSource:0}: Error finding container 306871116c73fa530ec9b78a923c016e2124ec686a28f07779d4ea9850098a86: Status 404 returned error can't find the container with id 306871116c73fa530ec9b78a923c016e2124ec686a28f07779d4ea9850098a86 Aug 05 18:20:09.832523 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.832486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" event={"ID":"01501edf-da8a-4d1a-9f55-90543201c206","Type":"ContainerStarted","Data":"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f"} Aug 05 18:20:09.834853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.832552 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:09.834853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.832577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" event={"ID":"01501edf-da8a-4d1a-9f55-90543201c206","Type":"ContainerStarted","Data":"306871116c73fa530ec9b78a923c016e2124ec686a28f07779d4ea9850098a86"} Aug 05 18:20:09.834853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.834000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" event={"ID":"f8fca617-a61f-4da2-9511-6e5e735bb56b","Type":"ContainerStarted","Data":"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225"} Aug 05 18:20:09.834853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.834039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" event={"ID":"f8fca617-a61f-4da2-9511-6e5e735bb56b","Type":"ContainerStarted","Data":"80ce9f849175c639d27ea3933dda49fe98c74bb9e2457ee750c77b663dc4cba6"} Aug 05 18:20:09.834853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.834066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:09.854599 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.854544 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" podStartSLOduration=1.85452745 podStartE2EDuration="1.85452745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:09.853280813 +0000 UTC m=+3466.414566428" watchObservedRunningTime="2026-08-05 18:20:09.85452745 +0000 UTC m=+3466.415813110" Aug 05 18:20:09.873301 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:09.873245 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" podStartSLOduration=1.873228574 podStartE2EDuration="1.873228574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:09.872073192 +0000 UTC m=+3466.433358806" watchObservedRunningTime="2026-08-05 18:20:09.873228574 +0000 UTC m=+3466.434514188" Aug 05 18:20:12.915067 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:12.915030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:20:20.841083 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:20.841052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:20.841559 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:20.841362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:38.365493 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.365462 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:38.365969 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.365698 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" podUID="f8fca617-a61f-4da2-9511-6e5e735bb56b" containerName="payload-processing" containerID="cri-o://5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225" gracePeriod=30 Aug 05 18:20:38.374061 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.374032 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:38.374287 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.374242 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" podUID="01501edf-da8a-4d1a-9f55-90543201c206" containerName="payload-pre-processing" containerID="cri-o://481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f" gracePeriod=30 Aug 05 18:20:38.640522 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.640489 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:38.643273 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.643253 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:38.780125 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d4sk\" (UniqueName: \"kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk\") pod \"f8fca617-a61f-4da2-9511-6e5e735bb56b\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " Aug 05 18:20:38.780125 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume\") pod \"01501edf-da8a-4d1a-9f55-90543201c206\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " Aug 05 18:20:38.780430 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume\") pod \"f8fca617-a61f-4da2-9511-6e5e735bb56b\" (UID: \"f8fca617-a61f-4da2-9511-6e5e735bb56b\") " Aug 05 18:20:38.780430 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780244 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txms8\" (UniqueName: \"kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8\") pod \"01501edf-da8a-4d1a-9f55-90543201c206\" (UID: \"01501edf-da8a-4d1a-9f55-90543201c206\") " Aug 05 18:20:38.780549 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01501edf-da8a-4d1a-9f55-90543201c206" (UID: "01501edf-da8a-4d1a-9f55-90543201c206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:20:38.780634 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.780609 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8fca617-a61f-4da2-9511-6e5e735bb56b" (UID: "f8fca617-a61f-4da2-9511-6e5e735bb56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:20:38.782337 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.782306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8" (OuterVolumeSpecName: "kube-api-access-txms8") pod "01501edf-da8a-4d1a-9f55-90543201c206" (UID: "01501edf-da8a-4d1a-9f55-90543201c206"). InnerVolumeSpecName "kube-api-access-txms8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:20:38.782403 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.782362 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk" (OuterVolumeSpecName: "kube-api-access-2d4sk") pod "f8fca617-a61f-4da2-9511-6e5e735bb56b" (UID: "f8fca617-a61f-4da2-9511-6e5e735bb56b"). InnerVolumeSpecName "kube-api-access-2d4sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:20:38.881459 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.881387 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4sk\" (UniqueName: \"kubernetes.io/projected/f8fca617-a61f-4da2-9511-6e5e735bb56b-kube-api-access-2d4sk\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:20:38.881459 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.881412 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01501edf-da8a-4d1a-9f55-90543201c206-plugins-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:20:38.881459 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.881422 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fca617-a61f-4da2-9511-6e5e735bb56b-plugins-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:20:38.881459 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.881432 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txms8\" (UniqueName: \"kubernetes.io/projected/01501edf-da8a-4d1a-9f55-90543201c206-kube-api-access-txms8\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:20:38.989213 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.989181 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8fca617-a61f-4da2-9511-6e5e735bb56b" containerID="5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225" exitCode=0 Aug 05 18:20:38.989426 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.989232 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" Aug 05 18:20:38.989426 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.989274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" event={"ID":"f8fca617-a61f-4da2-9511-6e5e735bb56b","Type":"ContainerDied","Data":"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225"} Aug 05 18:20:38.989426 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.989354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2" event={"ID":"f8fca617-a61f-4da2-9511-6e5e735bb56b","Type":"ContainerDied","Data":"80ce9f849175c639d27ea3933dda49fe98c74bb9e2457ee750c77b663dc4cba6"} Aug 05 18:20:38.989426 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.989387 2576 scope.go:117] "RemoveContainer" containerID="5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225" Aug 05 18:20:38.990638 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.990620 2576 generic.go:356] "Generic (PLEG): container finished" podID="01501edf-da8a-4d1a-9f55-90543201c206" containerID="481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f" exitCode=0 Aug 05 18:20:38.990714 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.990647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" event={"ID":"01501edf-da8a-4d1a-9f55-90543201c206","Type":"ContainerDied","Data":"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f"} Aug 05 18:20:38.990714 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.990677 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" Aug 05 18:20:38.990714 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.990694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb" event={"ID":"01501edf-da8a-4d1a-9f55-90543201c206","Type":"ContainerDied","Data":"306871116c73fa530ec9b78a923c016e2124ec686a28f07779d4ea9850098a86"} Aug 05 18:20:38.998315 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.998294 2576 scope.go:117] "RemoveContainer" containerID="5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225" Aug 05 18:20:38.998621 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:20:38.998602 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225\": container with ID starting with 5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225 not found: ID does not exist" containerID="5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225" Aug 05 18:20:38.998708 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.998633 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225"} err="failed to get container status \"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225\": rpc error: code = NotFound desc = could not find container \"5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225\": container with ID starting with 5a087c3a240964adf0faa44d68c2e26febc94f295d8ffc9e11e113f6b6433225 not found: ID does not exist" Aug 05 18:20:38.998708 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:38.998656 2576 scope.go:117] "RemoveContainer" containerID="481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f" Aug 05 18:20:39.006984 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.006966 2576 scope.go:117] "RemoveContainer" containerID="481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f" Aug 05 18:20:39.007215 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:20:39.007196 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f\": container with ID starting with 481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f not found: ID does not exist" containerID="481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f" Aug 05 18:20:39.007265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.007220 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f"} err="failed to get container status \"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f\": rpc error: code = NotFound desc = could not find container \"481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f\": container with ID starting with 481558708d561e5790991637b5ca677e7fe6f0f035377bf7b2e3fb6d1234ed2f not found: ID does not exist" Aug 05 18:20:39.023361 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.023337 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:39.030869 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.030839 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2696ea-f5999b696-mjzpb"] Aug 05 18:20:39.051609 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.051578 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:39.058014 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.057992 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2696ea-847dcd5b74-hgdj2"] Aug 05 18:20:39.979618 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.979588 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01501edf-da8a-4d1a-9f55-90543201c206" path="/var/lib/kubelet/pods/01501edf-da8a-4d1a-9f55-90543201c206/volumes" Aug 05 18:20:39.980001 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:39.979950 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8fca617-a61f-4da2-9511-6e5e735bb56b" path="/var/lib/kubelet/pods/f8fca617-a61f-4da2-9511-6e5e735bb56b/volumes" Aug 05 18:20:41.612895 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:41.612860 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:20:49.400835 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.400762 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp"] Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401160 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fca617-a61f-4da2-9511-6e5e735bb56b" containerName="payload-processing" Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401171 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fca617-a61f-4da2-9511-6e5e735bb56b" containerName="payload-processing" Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401187 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01501edf-da8a-4d1a-9f55-90543201c206" containerName="payload-pre-processing" Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401192 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01501edf-da8a-4d1a-9f55-90543201c206" containerName="payload-pre-processing" Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401252 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8fca617-a61f-4da2-9511-6e5e735bb56b" containerName="payload-processing" Aug 05 18:20:49.401270 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.401259 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01501edf-da8a-4d1a-9f55-90543201c206" containerName="payload-pre-processing" Aug 05 18:20:49.406017 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.405997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:49.408362 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.408341 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzbbt\"" Aug 05 18:20:49.413624 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.413580 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp"] Aug 05 18:20:49.465468 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.465441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp\" (UID: \"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:49.566793 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.566765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp\" (UID: \"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:49.567351 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.567314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp\" (UID: \"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:49.717758 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.717728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:49.852543 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:49.852515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp"] Aug 05 18:20:49.853303 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:20:49.853279 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3635632d_5c41_4c0f_8cb2_19bccb1e2cb3.slice/crio-fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb WatchSource:0}: Error finding container fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb: Status 404 returned error can't find the container with id fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb Aug 05 18:20:50.048658 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:50.048575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" event={"ID":"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3","Type":"ContainerStarted","Data":"fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb"} Aug 05 18:20:53.067488 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:53.067453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" event={"ID":"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3","Type":"ContainerStarted","Data":"6094dfcab1e8040fd03f1d5f680513e18ff8c90da67b719a2e9afadae9abee41"} Aug 05 18:20:53.087182 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:53.087127 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" podStartSLOduration=1.222298014 podStartE2EDuration="4.087110758s" totalImagesPullingTime="2.864812744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:53.084509487 +0000 UTC m=+3509.645795102" watchObservedRunningTime="2026-08-05 18:20:53.087110758 +0000 UTC m=+3509.648396371" Aug 05 18:20:54.074261 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:54.074225 2576 generic.go:356] "Generic (PLEG): container finished" podID="3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" containerID="6094dfcab1e8040fd03f1d5f680513e18ff8c90da67b719a2e9afadae9abee41" exitCode=0 Aug 05 18:20:54.074656 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:54.074274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" event={"ID":"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3","Type":"ContainerDied","Data":"6094dfcab1e8040fd03f1d5f680513e18ff8c90da67b719a2e9afadae9abee41"} Aug 05 18:20:55.214333 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:55.214299 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:55.318518 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:55.318489 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca\") pod \"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3\" (UID: \"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3\") " Aug 05 18:20:55.318815 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:55.318793 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" (UID: "3635632d-5c41-4c0f-8cb2-19bccb1e2cb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:20:55.419116 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:55.419050 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:20:56.085954 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:56.085923 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" Aug 05 18:20:56.086130 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:56.085964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp" event={"ID":"3635632d-5c41-4c0f-8cb2-19bccb1e2cb3","Type":"ContainerDied","Data":"fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb"} Aug 05 18:20:56.086130 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:20:56.086016 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb53abce13df86aa708a4775c7abf7baa468c5b6906503835fdd43a6a2e37ceb" Aug 05 18:21:03.141964 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:03.141930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:21:05.077013 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.076974 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:21:05.077520 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.077224 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" podUID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" containerName="payload-processing" containerID="cri-o://083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060" gracePeriod=30 Aug 05 18:21:05.244965 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.244928 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp"] Aug 05 18:21:05.248097 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.248068 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6427fe-hprdp"] Aug 05 18:21:05.336536 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.336471 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:21:05.414495 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.414464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6srqv\" (UniqueName: \"kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv\") pod \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " Aug 05 18:21:05.414663 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.414548 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume\") pod \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\" (UID: \"4ac873ca-d817-4f60-8d92-a2ac5bd683ea\") " Aug 05 18:21:05.414943 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.414921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ac873ca-d817-4f60-8d92-a2ac5bd683ea" (UID: "4ac873ca-d817-4f60-8d92-a2ac5bd683ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:21:05.416618 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.416595 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv" (OuterVolumeSpecName: "kube-api-access-6srqv") pod "4ac873ca-d817-4f60-8d92-a2ac5bd683ea" (UID: "4ac873ca-d817-4f60-8d92-a2ac5bd683ea"). InnerVolumeSpecName "kube-api-access-6srqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:21:05.515460 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.515422 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srqv\" (UniqueName: \"kubernetes.io/projected/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-kube-api-access-6srqv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:05.515460 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.515453 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac873ca-d817-4f60-8d92-a2ac5bd683ea-plugins-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:05.978260 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:05.978226 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" path="/var/lib/kubelet/pods/3635632d-5c41-4c0f-8cb2-19bccb1e2cb3/volumes" Aug 05 18:21:06.135917 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.135876 2576 generic.go:356] "Generic (PLEG): container finished" podID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" containerID="083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060" exitCode=0 Aug 05 18:21:06.136383 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.135924 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" Aug 05 18:21:06.136383 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.135960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" event={"ID":"4ac873ca-d817-4f60-8d92-a2ac5bd683ea","Type":"ContainerDied","Data":"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060"} Aug 05 18:21:06.136383 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.136020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc" event={"ID":"4ac873ca-d817-4f60-8d92-a2ac5bd683ea","Type":"ContainerDied","Data":"fcf642a6eecb0d7ed5b19568c0f78d15b75f005f81f62849c0f4d2dc5f8ecde4"} Aug 05 18:21:06.136383 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.136045 2576 scope.go:117] "RemoveContainer" containerID="083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060" Aug 05 18:21:06.145365 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.145343 2576 scope.go:117] "RemoveContainer" containerID="083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060" Aug 05 18:21:06.145644 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:21:06.145624 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060\": container with ID starting with 083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060 not found: ID does not exist" containerID="083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060" Aug 05 18:21:06.145732 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.145652 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060"} err="failed to get container status \"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060\": rpc error: code = NotFound desc = could not find container \"083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060\": container with ID starting with 083a318a819d64e3ef39e7d27b93f790843017e36eb6cdcdda0fe1cdf3699060 not found: ID does not exist" Aug 05 18:21:06.151535 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.151506 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:21:06.155604 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:06.155578 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6427fe-7d47786954-9hszc"] Aug 05 18:21:07.978744 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:07.978713 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" path="/var/lib/kubelet/pods/4ac873ca-d817-4f60-8d92-a2ac5bd683ea/volumes" Aug 05 18:21:10.616391 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:10.616356 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:21:10.944401 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:10.944360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:21:25.638470 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:25.638409 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" podUID="5859b046-9924-4df1-8269-d2c2e87db3d2" containerName="main" containerID="cri-o://632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced" gracePeriod=45 Aug 05 18:21:25.889354 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:25.889259 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:21:26.012003 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.011971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012003 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg2t4\" (UniqueName: \"kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012227 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache\") pod \"5859b046-9924-4df1-8269-d2c2e87db3d2\" (UID: \"5859b046-9924-4df1-8269-d2c2e87db3d2\") " Aug 05 18:21:26.012509 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012392 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:21:26.012509 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012427 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home" (OuterVolumeSpecName: "home") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:21:26.012587 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012534 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:21:26.012666 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012650 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-tmp-dir\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.012704 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012668 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-home\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.012704 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.012677 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.014380 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.014353 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4" (OuterVolumeSpecName: "kube-api-access-hg2t4") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "kube-api-access-hg2t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:21:26.014660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.014633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:21:26.014660 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.014633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm" (OuterVolumeSpecName: "dshm") pod "5859b046-9924-4df1-8269-d2c2e87db3d2" (UID: "5859b046-9924-4df1-8269-d2c2e87db3d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:21:26.113538 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.113503 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5859b046-9924-4df1-8269-d2c2e87db3d2-dshm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.113538 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.113530 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2t4\" (UniqueName: \"kubernetes.io/projected/5859b046-9924-4df1-8269-d2c2e87db3d2-kube-api-access-hg2t4\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.113538 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.113542 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5859b046-9924-4df1-8269-d2c2e87db3d2-tls-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:21:26.246313 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.246282 2576 generic.go:356] "Generic (PLEG): container finished" podID="5859b046-9924-4df1-8269-d2c2e87db3d2" containerID="632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced" exitCode=0 Aug 05 18:21:26.246482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.246360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" event={"ID":"5859b046-9924-4df1-8269-d2c2e87db3d2","Type":"ContainerDied","Data":"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced"} Aug 05 18:21:26.246482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.246404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" event={"ID":"5859b046-9924-4df1-8269-d2c2e87db3d2","Type":"ContainerDied","Data":"592304fb31f8c858f16dc0807998d89db3cc85f3617a527e4bb21902e0fcfc65"} Aug 05 18:21:26.246482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.246422 2576 scope.go:117] "RemoveContainer" containerID="632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced" Aug 05 18:21:26.246482 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.246360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk" Aug 05 18:21:26.255752 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.255740 2576 scope.go:117] "RemoveContainer" containerID="632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced" Aug 05 18:21:26.256002 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:21:26.255985 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced\": container with ID starting with 632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced not found: ID does not exist" containerID="632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced" Aug 05 18:21:26.256078 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.256007 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced"} err="failed to get container status \"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced\": rpc error: code = NotFound desc = could not find container \"632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced\": container with ID starting with 632c132bf3da2fa564fcdb00f42cd0c528caaaf268e7cf7283254568173c6ced not found: ID does not exist" Aug 05 18:21:26.267726 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.267704 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:21:26.271622 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:26.271597 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6427fe/ipp-route-6427fe-kserve-bf8977d4c-n7wdk"] Aug 05 18:21:27.977727 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:21:27.977692 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5859b046-9924-4df1-8269-d2c2e87db3d2" path="/var/lib/kubelet/pods/5859b046-9924-4df1-8269-d2c2e87db3d2/volumes" Aug 05 18:22:18.528246 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528146 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp"] Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528585 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" containerName="payload-processing" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528599 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" containerName="payload-processing" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528606 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" containerName="revoke-keys" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528613 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" containerName="revoke-keys" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528626 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5859b046-9924-4df1-8269-d2c2e87db3d2" containerName="main" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528634 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5859b046-9924-4df1-8269-d2c2e87db3d2" containerName="main" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528698 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3635632d-5c41-4c0f-8cb2-19bccb1e2cb3" containerName="revoke-keys" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528707 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5859b046-9924-4df1-8269-d2c2e87db3d2" containerName="main" Aug 05 18:22:18.528841 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.528714 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac873ca-d817-4f60-8d92-a2ac5bd683ea" containerName="payload-processing" Aug 05 18:22:18.531834 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.531817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:18.534399 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.534377 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzbbt\"" Aug 05 18:22:18.546815 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.546776 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp"] Aug 05 18:22:18.596160 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.596126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp\" (UID: \"50fcaf9b-74dc-448d-89cb-67019653becd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:18.697296 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.697257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp\" (UID: \"50fcaf9b-74dc-448d-89cb-67019653becd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:18.697911 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.697888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp\" (UID: \"50fcaf9b-74dc-448d-89cb-67019653becd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:18.847866 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.847775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:18.988550 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:18.988523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp"] Aug 05 18:22:18.991091 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:22:18.991063 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fcaf9b_74dc_448d_89cb_67019653becd.slice/crio-29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9 WatchSource:0}: Error finding container 29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9: Status 404 returned error can't find the container with id 29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9 Aug 05 18:22:19.506573 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:19.506540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" event={"ID":"50fcaf9b-74dc-448d-89cb-67019653becd","Type":"ContainerStarted","Data":"af3790de3748a235ca26ad45ff21dd349f5f8b048e6b4181fa475bb45f10c206"} Aug 05 18:22:19.506755 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:19.506593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" event={"ID":"50fcaf9b-74dc-448d-89cb-67019653becd","Type":"ContainerStarted","Data":"29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9"} Aug 05 18:22:19.523272 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:19.523221 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" podStartSLOduration=1.5232058990000001 podStartE2EDuration="1.523205899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:22:19.522809561 +0000 UTC m=+3596.084095187" watchObservedRunningTime="2026-08-05 18:22:19.523205899 +0000 UTC m=+3596.084491512" Aug 05 18:22:20.512611 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:20.512575 2576 generic.go:356] "Generic (PLEG): container finished" podID="50fcaf9b-74dc-448d-89cb-67019653becd" containerID="af3790de3748a235ca26ad45ff21dd349f5f8b048e6b4181fa475bb45f10c206" exitCode=0 Aug 05 18:22:20.512991 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:20.512655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" event={"ID":"50fcaf9b-74dc-448d-89cb-67019653becd","Type":"ContainerDied","Data":"af3790de3748a235ca26ad45ff21dd349f5f8b048e6b4181fa475bb45f10c206"} Aug 05 18:22:21.666809 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:21.666784 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:21.727221 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:21.727188 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca\") pod \"50fcaf9b-74dc-448d-89cb-67019653becd\" (UID: \"50fcaf9b-74dc-448d-89cb-67019653becd\") " Aug 05 18:22:21.727550 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:21.727526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50fcaf9b-74dc-448d-89cb-67019653becd" (UID: "50fcaf9b-74dc-448d-89cb-67019653becd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:22:21.828714 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:21.828632 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcaf9b-74dc-448d-89cb-67019653becd-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:22.524148 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:22.524120 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" Aug 05 18:22:22.524360 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:22.524157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp" event={"ID":"50fcaf9b-74dc-448d-89cb-67019653becd","Type":"ContainerDied","Data":"29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9"} Aug 05 18:22:22.524360 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:22.524206 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29517218ec469dbaf98f8c15f36b4fb0d81b192e27cd42909a75313c86ab23c9" Aug 05 18:22:24.154824 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:24.154794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:22:24.167018 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:24.166995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:22:29.165752 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.165716 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:22:29.166227 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.166035 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" podUID="db4b4db0-826c-43bf-ac08-809de254ebe9" containerName="maas-api" containerID="cri-o://0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095" gracePeriod=30 Aug 05 18:22:29.230247 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.230207 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:22:29.232139 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.231894 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" podUID="281cc71c-512b-44e5-9a7d-8c734ad74909" containerName="payload-processing" containerID="cri-o://c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07" gracePeriod=30 Aug 05 18:22:29.236839 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.236753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:22:29.237019 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.236948 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" podUID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" containerName="payload-pre-processing" containerID="cri-o://8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd" gracePeriod=30 Aug 05 18:22:29.431442 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.431415 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp"] Aug 05 18:22:29.444920 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.444887 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a3125a-4zzzp"] Aug 05 18:22:29.492242 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.492218 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:22:29.512828 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.512800 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:22:29.524197 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.524178 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:22:29.561302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.561271 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4b4db0-826c-43bf-ac08-809de254ebe9" containerID="0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095" exitCode=0 Aug 05 18:22:29.561302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.561302 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" Aug 05 18:22:29.561565 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.561365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" event={"ID":"db4b4db0-826c-43bf-ac08-809de254ebe9","Type":"ContainerDied","Data":"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095"} Aug 05 18:22:29.561565 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.561417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl" event={"ID":"db4b4db0-826c-43bf-ac08-809de254ebe9","Type":"ContainerDied","Data":"dfe092dd75022350cf720bcaf968978ad4db4e148d0cc5aecd8282eabaa2b1ed"} Aug 05 18:22:29.561565 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.561451 2576 scope.go:117] "RemoveContainer" containerID="0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095" Aug 05 18:22:29.562657 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.562636 2576 generic.go:356] "Generic (PLEG): container finished" podID="281cc71c-512b-44e5-9a7d-8c734ad74909" containerID="c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07" exitCode=0 Aug 05 18:22:29.562771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.562678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" Aug 05 18:22:29.562771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.562720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" event={"ID":"281cc71c-512b-44e5-9a7d-8c734ad74909","Type":"ContainerDied","Data":"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07"} Aug 05 18:22:29.562771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.562762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm" event={"ID":"281cc71c-512b-44e5-9a7d-8c734ad74909","Type":"ContainerDied","Data":"2565890e80b134d8af45d77493e91d0dc34d0f53bf18460798bf5c402f334449"} Aug 05 18:22:29.564026 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.564004 2576 generic.go:356] "Generic (PLEG): container finished" podID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" containerID="8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd" exitCode=0 Aug 05 18:22:29.564149 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.564046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" Aug 05 18:22:29.564149 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.564086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" event={"ID":"e4b4df30-7995-4387-8a0a-e6749b3dc97f","Type":"ContainerDied","Data":"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd"} Aug 05 18:22:29.564149 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.564125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t" event={"ID":"e4b4df30-7995-4387-8a0a-e6749b3dc97f","Type":"ContainerDied","Data":"f714287f77a5658b1d9a9e2aa7f4ed45b020d8c93c79ff8ea2a614b8c766b0b2"} Aug 05 18:22:29.570182 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.570166 2576 scope.go:117] "RemoveContainer" containerID="0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095" Aug 05 18:22:29.570470 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:22:29.570443 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095\": container with ID starting with 0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095 not found: ID does not exist" containerID="0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095" Aug 05 18:22:29.570543 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.570505 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095"} err="failed to get container status \"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095\": rpc error: code = NotFound desc = could not find container \"0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095\": container with ID starting with 0e5f6882f1fdb3ad26c5aecf0262f6b92bf8af03b85f48a456f564510fe56095 not found: ID does not exist" Aug 05 18:22:29.570543 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.570528 2576 scope.go:117] "RemoveContainer" containerID="c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07" Aug 05 18:22:29.578036 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.578012 2576 scope.go:117] "RemoveContainer" containerID="c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07" Aug 05 18:22:29.578273 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:22:29.578255 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07\": container with ID starting with c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07 not found: ID does not exist" containerID="c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07" Aug 05 18:22:29.578353 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.578279 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07"} err="failed to get container status \"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07\": rpc error: code = NotFound desc = could not find container \"c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07\": container with ID starting with c487afd101edfc2a36d76561ff86df2e4bb5116e5d9c075207d612d0f6758f07 not found: ID does not exist" Aug 05 18:22:29.578353 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.578296 2576 scope.go:117] "RemoveContainer" containerID="8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd" Aug 05 18:22:29.585552 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.585535 2576 scope.go:117] "RemoveContainer" containerID="8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd" Aug 05 18:22:29.585787 ip-10-0-131-14 kubenswrapper[2576]: E0805 18:22:29.585769 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd\": container with ID starting with 8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd not found: ID does not exist" containerID="8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd" Aug 05 18:22:29.585843 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.585793 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd"} err="failed to get container status \"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd\": rpc error: code = NotFound desc = could not find container \"8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd\": container with ID starting with 8ef90124b0ed1a81dfebcf287a1925b8a6fcf14000885908e7d8ee17cdd967cd not found: ID does not exist" Aug 05 18:22:29.597110 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls\") pod \"db4b4db0-826c-43bf-ac08-809de254ebe9\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " Aug 05 18:22:29.597185 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597126 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume\") pod \"281cc71c-512b-44e5-9a7d-8c734ad74909\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " Aug 05 18:22:29.597185 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gs2c\" (UniqueName: \"kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c\") pod \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " Aug 05 18:22:29.597281 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw2w6\" (UniqueName: \"kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6\") pod \"db4b4db0-826c-43bf-ac08-809de254ebe9\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " Aug 05 18:22:29.597281 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7h5j\" (UniqueName: \"kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j\") pod \"281cc71c-512b-44e5-9a7d-8c734ad74909\" (UID: \"281cc71c-512b-44e5-9a7d-8c734ad74909\") " Aug 05 18:22:29.597412 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca\") pod \"db4b4db0-826c-43bf-ac08-809de254ebe9\" (UID: \"db4b4db0-826c-43bf-ac08-809de254ebe9\") " Aug 05 18:22:29.597412 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597354 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume\") pod \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\" (UID: \"e4b4df30-7995-4387-8a0a-e6749b3dc97f\") " Aug 05 18:22:29.597754 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597514 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "281cc71c-512b-44e5-9a7d-8c734ad74909" (UID: "281cc71c-512b-44e5-9a7d-8c734ad74909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:22:29.597827 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db4b4db0-826c-43bf-ac08-809de254ebe9" (UID: "db4b4db0-826c-43bf-ac08-809de254ebe9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:22:29.597827 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597753 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4b4df30-7995-4387-8a0a-e6749b3dc97f" (UID: "e4b4df30-7995-4387-8a0a-e6749b3dc97f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:22:29.597827 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.597755 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281cc71c-512b-44e5-9a7d-8c734ad74909-plugins-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.599252 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.599233 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db4b4db0-826c-43bf-ac08-809de254ebe9" (UID: "db4b4db0-826c-43bf-ac08-809de254ebe9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:22:29.599488 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.599458 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6" (OuterVolumeSpecName: "kube-api-access-fw2w6") pod "db4b4db0-826c-43bf-ac08-809de254ebe9" (UID: "db4b4db0-826c-43bf-ac08-809de254ebe9"). InnerVolumeSpecName "kube-api-access-fw2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:22:29.599672 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.599650 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c" (OuterVolumeSpecName: "kube-api-access-2gs2c") pod "e4b4df30-7995-4387-8a0a-e6749b3dc97f" (UID: "e4b4df30-7995-4387-8a0a-e6749b3dc97f"). InnerVolumeSpecName "kube-api-access-2gs2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:22:29.599731 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.599677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j" (OuterVolumeSpecName: "kube-api-access-f7h5j") pod "281cc71c-512b-44e5-9a7d-8c734ad74909" (UID: "281cc71c-512b-44e5-9a7d-8c734ad74909"). InnerVolumeSpecName "kube-api-access-f7h5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:22:29.698864 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698842 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4b4db0-826c-43bf-ac08-809de254ebe9-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.698978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698865 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b4df30-7995-4387-8a0a-e6749b3dc97f-plugins-config-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.698978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698882 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4b4db0-826c-43bf-ac08-809de254ebe9-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.698978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698897 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gs2c\" (UniqueName: \"kubernetes.io/projected/e4b4df30-7995-4387-8a0a-e6749b3dc97f-kube-api-access-2gs2c\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.698978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698908 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2w6\" (UniqueName: \"kubernetes.io/projected/db4b4db0-826c-43bf-ac08-809de254ebe9-kube-api-access-fw2w6\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.698978 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.698920 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7h5j\" (UniqueName: \"kubernetes.io/projected/281cc71c-512b-44e5-9a7d-8c734ad74909-kube-api-access-f7h5j\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:29.905748 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.905713 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:22:29.922707 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.922682 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3125a-6c89b9575-8wgsl"] Aug 05 18:22:29.940679 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.940651 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:22:29.949308 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.949251 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a3125a-7659868978-b6vgm"] Aug 05 18:22:29.964304 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.964280 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:22:29.969506 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.969487 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a3125a-74686fbdb8-9zt6t"] Aug 05 18:22:29.978464 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.978435 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281cc71c-512b-44e5-9a7d-8c734ad74909" path="/var/lib/kubelet/pods/281cc71c-512b-44e5-9a7d-8c734ad74909/volumes" Aug 05 18:22:29.978773 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.978760 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fcaf9b-74dc-448d-89cb-67019653becd" path="/var/lib/kubelet/pods/50fcaf9b-74dc-448d-89cb-67019653becd/volumes" Aug 05 18:22:29.979097 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.979086 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4b4db0-826c-43bf-ac08-809de254ebe9" path="/var/lib/kubelet/pods/db4b4db0-826c-43bf-ac08-809de254ebe9/volumes" Aug 05 18:22:29.979413 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:29.979401 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" path="/var/lib/kubelet/pods/e4b4df30-7995-4387-8a0a-e6749b3dc97f/volumes" Aug 05 18:22:31.416545 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:31.416513 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:22:36.777460 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.777427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7"] Aug 05 18:22:36.777954 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.777935 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281cc71c-512b-44e5-9a7d-8c734ad74909" containerName="payload-processing" Aug 05 18:22:36.778056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.777957 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="281cc71c-512b-44e5-9a7d-8c734ad74909" containerName="payload-processing" Aug 05 18:22:36.778056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.777974 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4b4db0-826c-43bf-ac08-809de254ebe9" containerName="maas-api" Aug 05 18:22:36.778056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.777982 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4b4db0-826c-43bf-ac08-809de254ebe9" containerName="maas-api" Aug 05 18:22:36.778056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778027 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fcaf9b-74dc-448d-89cb-67019653becd" containerName="revoke-keys" Aug 05 18:22:36.778056 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778036 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fcaf9b-74dc-448d-89cb-67019653becd" containerName="revoke-keys" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778070 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" containerName="payload-pre-processing" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778080 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" containerName="payload-pre-processing" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778178 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fcaf9b-74dc-448d-89cb-67019653becd" containerName="revoke-keys" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778191 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b4df30-7995-4387-8a0a-e6749b3dc97f" containerName="payload-pre-processing" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778200 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4b4db0-826c-43bf-ac08-809de254ebe9" containerName="maas-api" Aug 05 18:22:36.778302 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.778211 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="281cc71c-512b-44e5-9a7d-8c734ad74909" containerName="payload-processing" Aug 05 18:22:36.782742 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.782723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:36.785476 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.785457 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:22:36.785965 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.785945 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzbbt\"" Aug 05 18:22:36.791763 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.791737 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7"] Aug 05 18:22:36.857820 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.857794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7\" (UID: \"073db792-c748-4bbf-9e4e-610296a2a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:36.959031 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.958995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7\" (UID: \"073db792-c748-4bbf-9e4e-610296a2a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:36.959586 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:36.959570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7\" (UID: \"073db792-c748-4bbf-9e4e-610296a2a170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:37.094682 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:37.094601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:37.222787 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:37.222756 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7"] Aug 05 18:22:37.225675 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:22:37.225652 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073db792_c748_4bbf_9e4e_610296a2a170.slice/crio-9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9 WatchSource:0}: Error finding container 9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9: Status 404 returned error can't find the container with id 9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9 Aug 05 18:22:37.611014 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:37.610978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" event={"ID":"073db792-c748-4bbf-9e4e-610296a2a170","Type":"ContainerStarted","Data":"72e8a0cc4e0f3189501f5d6b406dc8b3ea98bd9e8e9133b83bd898a74923bb8a"} Aug 05 18:22:37.611159 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:37.611036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" event={"ID":"073db792-c748-4bbf-9e4e-610296a2a170","Type":"ContainerStarted","Data":"9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9"} Aug 05 18:22:37.626766 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:37.626431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" podStartSLOduration=1.626413004 podStartE2EDuration="1.626413004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:22:37.62601157 +0000 UTC m=+3614.187297198" watchObservedRunningTime="2026-08-05 18:22:37.626413004 +0000 UTC m=+3614.187698618" Aug 05 18:22:38.617446 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:38.617410 2576 generic.go:356] "Generic (PLEG): container finished" podID="073db792-c748-4bbf-9e4e-610296a2a170" containerID="72e8a0cc4e0f3189501f5d6b406dc8b3ea98bd9e8e9133b83bd898a74923bb8a" exitCode=0 Aug 05 18:22:38.617820 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:38.617501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" event={"ID":"073db792-c748-4bbf-9e4e-610296a2a170","Type":"ContainerDied","Data":"72e8a0cc4e0f3189501f5d6b406dc8b3ea98bd9e8e9133b83bd898a74923bb8a"} Aug 05 18:22:39.760680 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:39.760654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:39.884571 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:39.884480 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca\") pod \"073db792-c748-4bbf-9e4e-610296a2a170\" (UID: \"073db792-c748-4bbf-9e4e-610296a2a170\") " Aug 05 18:22:39.884875 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:39.884848 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "073db792-c748-4bbf-9e4e-610296a2a170" (UID: "073db792-c748-4bbf-9e4e-610296a2a170"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:22:39.985229 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:39.985182 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073db792-c748-4bbf-9e4e-610296a2a170-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Aug 05 18:22:40.628772 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:40.628742 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" Aug 05 18:22:40.628772 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:40.628761 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7" event={"ID":"073db792-c748-4bbf-9e4e-610296a2a170","Type":"ContainerDied","Data":"9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9"} Aug 05 18:22:40.628981 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:40.628802 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7c8c357795b2b1503f3b302641de92bd16a818b3556c2471afc875300e06f9" Aug 05 18:22:47.680206 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:47.680161 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7"] Aug 05 18:22:47.683046 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:47.683020 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ba69fc-hfgj7"] Aug 05 18:22:47.978264 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:47.978218 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="073db792-c748-4bbf-9e4e-610296a2a170" path="/var/lib/kubelet/pods/073db792-c748-4bbf-9e4e-610296a2a170/volumes" Aug 05 18:22:49.409743 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:22:49.409714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:23:07.618019 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:07.617985 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2xtrv"] Aug 05 18:23:19.447403 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:19.447368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cc69f54b9-rvn7q_61b6834f-a1f3-4f32-a3f3-ab43e3d614c0/manager/0.log" Aug 05 18:23:19.454931 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:19.454912 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cc69f54b9-rvn7q_61b6834f-a1f3-4f32-a3f3-ab43e3d614c0/copy-manifests/0.log" Aug 05 18:23:21.317818 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:21.317786 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-f6wcv_9c4bbd07-0cd9-4952-a391-950266e993a8/kuadrant-console-plugin/0.log" Aug 05 18:23:21.534725 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:21.534688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-2xtrv_b522ddfc-4566-4765-bee8-e8c67be975b0/limitador/0.log" Aug 05 18:23:22.228966 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:22.228937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746fd96bc4-5s4vp_aefb29d2-b574-40fb-82ab-e833035c57bb/kube-auth-proxy/0.log" Aug 05 18:23:22.795829 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:22.795801 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8cf7bccb4-x647z_a61e86fd-ff4d-416b-b21a-6fd2cc238f89/router/0.log" Aug 05 18:23:23.127402 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:23.127328 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-d5s78_98443830-bc9b-475b-97f2-e546ad595272/main/0.log" Aug 05 18:23:23.566503 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:23.566477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-x9jtk_652b0a35-339d-4dcf-8f98-bacbc1c480db/main/0.log" Aug 05 18:23:23.675374 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:23.675344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pzwqv_9942cc94-73b9-4f47-87e2-ed0adbfd949d/main/0.log" Aug 05 18:23:32.356446 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:32.356410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cc44b_67a1d7d7-54c6-4f96-8633-3409d07c9a28/global-pull-secret-syncer/0.log" Aug 05 18:23:32.597976 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:32.597941 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-thxcm_e1e5680e-ed52-40a5-8225-973f315728b2/konnectivity-agent/0.log" Aug 05 18:23:32.624208 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:32.624128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-14.ec2.internal_1c415e6671b04646ed97d73439cdf78e/haproxy/0.log" Aug 05 18:23:37.851967 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:37.851936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-f6wcv_9c4bbd07-0cd9-4952-a391-950266e993a8/kuadrant-console-plugin/0.log" Aug 05 18:23:37.978183 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:37.978155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-2xtrv_b522ddfc-4566-4765-bee8-e8c67be975b0/limitador/0.log" Aug 05 18:23:39.655534 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:39.655504 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-79qx2_487ef48c-f468-461c-b45a-f001a69e0edd/cluster-monitoring-operator/0.log" Aug 05 18:23:39.684332 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:39.684288 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9b9mj_0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f/kube-state-metrics/0.log" Aug 05 18:23:39.714138 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:39.714114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9b9mj_0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f/kube-rbac-proxy-main/0.log" Aug 05 18:23:39.744540 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:39.744515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9b9mj_0179fcf5-fbed-495b-b6dc-63e7cdcbfd4f/kube-rbac-proxy-self/0.log" Aug 05 18:23:39.783133 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:39.783099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6494764dfb-b5v66_8d35efb3-2d3d-4f49-81de-6666003cf733/metrics-server/0.log" Aug 05 18:23:40.057671 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.057647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcrll_269aee79-9d73-44c4-b971-4ee0012c9a95/node-exporter/0.log" Aug 05 18:23:40.081155 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.081122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcrll_269aee79-9d73-44c4-b971-4ee0012c9a95/kube-rbac-proxy/0.log" Aug 05 18:23:40.102286 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.102263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hcrll_269aee79-9d73-44c4-b971-4ee0012c9a95/init-textfile/0.log" Aug 05 18:23:40.217098 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.217072 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-x8qzr_c7bb9561-ba26-48ae-b21a-3f7a3645c91a/kube-rbac-proxy-main/0.log" Aug 05 18:23:40.245744 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.245717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-x8qzr_c7bb9561-ba26-48ae-b21a-3f7a3645c91a/kube-rbac-proxy-self/0.log" Aug 05 18:23:40.269683 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.269654 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-x8qzr_c7bb9561-ba26-48ae-b21a-3f7a3645c91a/openshift-state-metrics/0.log" Aug 05 18:23:40.310579 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.310494 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/prometheus/0.log" Aug 05 18:23:40.334565 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.334533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/config-reloader/0.log" Aug 05 18:23:40.361025 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.360999 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/thanos-sidecar/0.log" Aug 05 18:23:40.385493 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.385465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/kube-rbac-proxy-web/0.log" Aug 05 18:23:40.410442 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.410418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/kube-rbac-proxy/0.log" Aug 05 18:23:40.437124 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.437100 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/kube-rbac-proxy-thanos/0.log" Aug 05 18:23:40.458668 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.458646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225af48d-5824-48a0-931d-6259cf4ac5da/init-config-reloader/0.log" Aug 05 18:23:40.543108 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:40.543080 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-9264f_95b0f1fc-6504-4228-938b-2dbd60007f66/prometheus-operator-admission-webhook/0.log" Aug 05 18:23:42.286035 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.285997 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx"] Aug 05 18:23:42.286456 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.286377 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073db792-c748-4bbf-9e4e-610296a2a170" containerName="revoke-keys" Aug 05 18:23:42.286456 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.286390 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="073db792-c748-4bbf-9e4e-610296a2a170" containerName="revoke-keys" Aug 05 18:23:42.286530 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.286462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="073db792-c748-4bbf-9e4e-610296a2a170" containerName="revoke-keys" Aug 05 18:23:42.289474 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.289454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.291754 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.291729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dz5d\"/\"openshift-service-ca.crt\"" Aug 05 18:23:42.292599 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.292586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dz5d\"/\"kube-root-ca.crt\"" Aug 05 18:23:42.292665 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.292602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8dz5d\"/\"default-dockercfg-9zj9h\"" Aug 05 18:23:42.297032 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.297000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx"] Aug 05 18:23:42.348903 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.348868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrv9c\" (UniqueName: \"kubernetes.io/projected/cf1898b1-a894-4ec2-ae43-4c096e0fa056-kube-api-access-qrv9c\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.349093 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.348918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-proc\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.349093 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.348996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-lib-modules\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.349093 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.349033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-sys\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.349093 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.349062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-podres\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450305 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrv9c\" (UniqueName: \"kubernetes.io/projected/cf1898b1-a894-4ec2-ae43-4c096e0fa056-kube-api-access-qrv9c\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450537 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-proc\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450537 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-lib-modules\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450537 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-sys\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450537 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-podres\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450537 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-proc\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450743 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-sys\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450743 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-lib-modules\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.450743 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.450658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf1898b1-a894-4ec2-ae43-4c096e0fa056-podres\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.457611 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.457587 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrv9c\" (UniqueName: \"kubernetes.io/projected/cf1898b1-a894-4ec2-ae43-4c096e0fa056-kube-api-access-qrv9c\") pod \"perf-node-gather-daemonset-b9xjx\" (UID: \"cf1898b1-a894-4ec2-ae43-4c096e0fa056\") " pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.600687 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.600595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.671475 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.668094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/2.log" Aug 05 18:23:42.687110 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.687068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xp7c7_15dd0baa-aeab-4a15-9f72-d595f6b9099d/console-operator/3.log" Aug 05 18:23:42.762156 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.762084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx"] Aug 05 18:23:42.764849 ip-10-0-131-14 kubenswrapper[2576]: W0805 18:23:42.764822 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcf1898b1_a894_4ec2_ae43_4c096e0fa056.slice/crio-31810ea6ccc1c6bcd8432d6462a1aad255d79e035b842b2d8560d22e954a4467 WatchSource:0}: Error finding container 31810ea6ccc1c6bcd8432d6462a1aad255d79e035b842b2d8560d22e954a4467: Status 404 returned error can't find the container with id 31810ea6ccc1c6bcd8432d6462a1aad255d79e035b842b2d8560d22e954a4467 Aug 05 18:23:42.933245 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.933144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" event={"ID":"cf1898b1-a894-4ec2-ae43-4c096e0fa056","Type":"ContainerStarted","Data":"feb571777db0027c227566527698e6b77721bebc9a6509723658f57ff32b7632"} Aug 05 18:23:42.933565 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.933540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" event={"ID":"cf1898b1-a894-4ec2-ae43-4c096e0fa056","Type":"ContainerStarted","Data":"31810ea6ccc1c6bcd8432d6462a1aad255d79e035b842b2d8560d22e954a4467"} Aug 05 18:23:42.934265 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.934244 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:42.952082 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:42.952036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" podStartSLOduration=0.952023631 podStartE2EDuration="952.023631ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 18:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:23:42.950089293 +0000 UTC m=+3679.511374906" watchObservedRunningTime="2026-08-05 18:23:42.952023631 +0000 UTC m=+3679.513309245" Aug 05 18:23:45.219291 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:45.219260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7sh6g_342804d1-3b24-4d69-8016-286343d5ba8b/dns/0.log" Aug 05 18:23:45.246216 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:45.246185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7sh6g_342804d1-3b24-4d69-8016-286343d5ba8b/kube-rbac-proxy/0.log" Aug 05 18:23:45.372054 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:45.372028 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh92q_544f2794-e894-49e8-b7e9-bb2b030fa972/dns-node-resolver/0.log" Aug 05 18:23:46.020348 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:46.020301 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfvz2_045408ce-93d2-435e-9f6c-8b9e0e454631/node-ca/0.log" Aug 05 18:23:47.296791 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:47.296755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746fd96bc4-5s4vp_aefb29d2-b574-40fb-82ab-e833035c57bb/kube-auth-proxy/0.log" Aug 05 18:23:47.472360 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:47.472313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8cf7bccb4-x647z_a61e86fd-ff4d-416b-b21a-6fd2cc238f89/router/0.log" Aug 05 18:23:48.305367 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.305315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cc456_e033614e-7c60-424e-b2f6-acfa70868a47/serve-healthcheck-canary/0.log" Aug 05 18:23:48.837651 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.837605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-kz674_c5c7e17b-705f-4cca-b600-ce211df20436/insights-operator/0.log" Aug 05 18:23:48.838520 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.838502 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-kz674_c5c7e17b-705f-4cca-b600-ce211df20436/insights-operator/1.log" Aug 05 18:23:48.935158 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.935127 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlxzj_8caead93-0332-4349-87cc-3cc043567e50/kube-rbac-proxy/0.log" Aug 05 18:23:48.948116 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.948087 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8dz5d/perf-node-gather-daemonset-b9xjx" Aug 05 18:23:48.959939 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.959913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlxzj_8caead93-0332-4349-87cc-3cc043567e50/exporter/0.log" Aug 05 18:23:48.987771 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:48.987743 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jlxzj_8caead93-0332-4349-87cc-3cc043567e50/extractor/0.log" Aug 05 18:23:52.475504 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:52.475474 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cc69f54b9-rvn7q_61b6834f-a1f3-4f32-a3f3-ab43e3d614c0/manager/0.log" Aug 05 18:23:52.496272 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:52.496195 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cc69f54b9-rvn7q_61b6834f-a1f3-4f32-a3f3-ab43e3d614c0/copy-manifests/0.log" Aug 05 18:23:53.192258 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:53.192232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746fd96bc4-5s4vp_aefb29d2-b574-40fb-82ab-e833035c57bb/kube-auth-proxy/0.log" Aug 05 18:23:53.298239 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:53.298207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8cf7bccb4-x647z_a61e86fd-ff4d-416b-b21a-6fd2cc238f89/router/0.log" Aug 05 18:23:53.749792 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:53.749768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57fd6d5d59-5nxq8_3da1ffa2-0217-4bdd-9864-5f01d55ee104/manager/0.log" Aug 05 18:23:54.831125 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:54.831095 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746fd96bc4-5s4vp_aefb29d2-b574-40fb-82ab-e833035c57bb/kube-auth-proxy/0.log" Aug 05 18:23:54.942853 ip-10-0-131-14 kubenswrapper[2576]: I0805 18:23:54.942820 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8cf7bccb4-x647z_a61e86fd-ff4d-416b-b21a-6fd2cc238f89/router/0.log"