Jul 17 18:09:53.073285 ip-10-0-136-225 systemd[1]: Starting Kubernetes Kubelet... Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.435153 2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439221 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439236 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439240 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439243 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439246 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439249 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439252 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.469369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439255 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439257 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439260 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439263 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439265 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439270 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439273 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439276 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439280 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439282 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439285 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439288 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439291 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439293 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439297 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439300 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439302 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439305 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439308 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439311 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.472759 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439313 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439316 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439318 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439321 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439324 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439327 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439329 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439332 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439335 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439337 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439340 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439342 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439348 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439352 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439355 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439358 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439362 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439365 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439369 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.473391 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439373 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439376 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439379 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439382 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439385 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439387 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439393 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439396 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439399 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439403 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439406 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439408 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439412 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439415 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439418 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439421 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439424 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439426 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439430 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.474673 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439434 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439436 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439439 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439442 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439444 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439447 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439450 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439452 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439456 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439459 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439462 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439466 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439468 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439471 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439474 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439477 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439479 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439482 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439485 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439488 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.475328 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439490 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439493 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439495 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439498 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439501 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439503 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439506 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439508 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439511 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439513 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439516 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439519 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439522 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.439525 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441063 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441071 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441075 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441078 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441081 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441083 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.476404 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441087 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441090 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441093 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441096 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441099 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441102 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441105 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441108 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441111 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441113 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441116 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441119 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441122 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441124 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441127 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441129 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441132 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441135 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441137 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441140 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.477267 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441142 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441144 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441148 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441151 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441154 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441157 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441160 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441162 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441165 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441168 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441174 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441176 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441179 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441182 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441185 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441187 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441190 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441193 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441197 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.481620 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441201 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441204 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441207 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441210 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441213 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441216 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441218 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441223 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441225 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441228 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441231 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441234 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441237 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441239 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441242 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441244 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441247 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441250 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441252 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441256 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.482286 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441258 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441261 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441264 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441268 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441270 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441274 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441276 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441279 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441281 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441284 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441286 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441289 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441291 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441293 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441296 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441298 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441301 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441303 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441306 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441310 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.483808 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441312 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441315 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441318 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441322 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441326 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441330 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441333 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441335 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441338 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441341 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441343 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441347 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441351 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441354 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441787 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441794 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441797 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441800 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441805 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.484471 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441808 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441811 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441815 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441818 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441821 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441824 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441826 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441830 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441832 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441835 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441837 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441839 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441843 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441846 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441848 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441851 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441854 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441857 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441859 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441862 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.485296 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441864 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441867 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441869 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441872 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441874 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441878 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441880 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441883 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441886 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441889 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441892 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441895 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441897 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441900 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441903 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441906 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441910 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441913 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441916 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.562365 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441918 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.514465 ip-10-0-136-225 systemd[1]: Started Kubernetes Kubelet. Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441921 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441924 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441927 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441929 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441934 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441936 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441939 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441942 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441945 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441947 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441950 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441953 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441955 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441958 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441960 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441963 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441965 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441968 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441971 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.563236 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441974 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.441995 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442000 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442003 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442007 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442011 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442014 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442016 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442019 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442021 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442024 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442027 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442029 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442032 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442034 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442037 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442040 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442044 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442048 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442051 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.564096 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442054 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442057 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442060 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442062 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442065 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442067 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442070 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442073 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442076 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442078 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442081 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442083 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442086 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442090 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442093 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442162 2587 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442169 2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442175 2587 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442180 2587 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442184 2587 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442188 2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 18:09:53.565695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442192 2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442197 2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442200 2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442203 2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442207 2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442211 2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442213 2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442217 2587 flags.go:64] FLAG: --cgroup-root="" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442221 2587 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442224 2587 flags.go:64] FLAG: --client-ca-file="" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442227 2587 flags.go:64] FLAG: --cloud-provider="external" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442230 2587 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442254 2587 flags.go:64] FLAG: --cluster-domain="" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442257 2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442261 2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442264 2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442268 2587 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442272 2587 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442275 2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442278 2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442281 2587 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442284 2587 flags.go:64] FLAG: --contention-profiling="false" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442287 2587 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442290 2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 18:09:53.566524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442295 2587 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442298 2587 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442303 2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442306 2587 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442309 2587 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442312 2587 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442315 2587 flags.go:64] FLAG: --enable-server="true" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442318 2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442322 2587 flags.go:64] FLAG: --event-burst="100" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442325 2587 flags.go:64] FLAG: --event-qps="50" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442328 2587 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442331 2587 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442334 2587 flags.go:64] FLAG: --eviction-hard="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442338 2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442341 2587 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442344 2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442349 2587 flags.go:64] FLAG: --eviction-soft="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442352 2587 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442355 2587 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442358 2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442360 2587 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442363 2587 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442366 2587 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442369 2587 flags.go:64] FLAG: --feature-gates="" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442373 2587 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 18:09:53.567381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442376 2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442379 2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442382 2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442385 2587 flags.go:64] FLAG: --healthz-port="10248" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442388 2587 flags.go:64] FLAG: --help="false" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442391 2587 flags.go:64] FLAG: --hostname-override="ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442394 2587 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442399 2587 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442403 2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442406 2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442410 2587 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442413 2587 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442416 2587 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442419 2587 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442422 2587 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442424 2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442428 2587 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442430 2587 flags.go:64] FLAG: --kube-reserved="" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442433 2587 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442436 2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442439 2587 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442442 2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442445 2587 flags.go:64] FLAG: --lock-file="" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442449 2587 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 18:09:53.568594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442452 2587 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442455 2587 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442460 2587 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442463 2587 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442466 2587 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442469 2587 flags.go:64] FLAG: --logging-format="text" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442472 2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442475 2587 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442478 2587 flags.go:64] FLAG: --manifest-url="" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442481 2587 flags.go:64] FLAG: --manifest-url-header="" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442486 2587 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442489 2587 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442493 2587 flags.go:64] FLAG: --max-pods="110" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442496 2587 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442499 2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442503 2587 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442506 2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442510 2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442513 2587 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442516 2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442524 2587 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442527 2587 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442530 2587 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 18:09:53.569498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442533 2587 flags.go:64] FLAG: --pod-cidr="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442536 2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442542 2587 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442545 2587 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442548 2587 flags.go:64] FLAG: --pods-per-core="0" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442551 2587 flags.go:64] FLAG: --port="10250" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442554 2587 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442557 2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063421d06b362447e" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442560 2587 flags.go:64] FLAG: --qos-reserved="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442564 2587 flags.go:64] FLAG: --read-only-port="10255" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442567 2587 flags.go:64] FLAG: --register-node="true" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442570 2587 flags.go:64] FLAG: --register-with-taints="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442574 2587 flags.go:64] FLAG: --registry-burst="10" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442577 2587 flags.go:64] FLAG: --registry-qps="5" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442580 2587 flags.go:64] FLAG: --reserved-cpus="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442583 2587 flags.go:64] FLAG: --reserved-memory="" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442587 2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442590 2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442593 2587 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442596 2587 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442599 2587 flags.go:64] FLAG: --runonce="false" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442602 2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442604 2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442608 2587 flags.go:64] FLAG: --seccomp-default="false" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442612 2587 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 18:09:53.570554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442615 2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442618 2587 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442622 2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442626 2587 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442629 2587 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442632 2587 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442635 2587 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442637 2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442640 2587 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442643 2587 flags.go:64] FLAG: --system-cgroups="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442647 2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442652 2587 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442655 2587 flags.go:64] FLAG: --tls-cert-file="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442657 2587 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442663 2587 flags.go:64] FLAG: --tls-min-version="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442666 2587 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442671 2587 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442674 2587 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442677 2587 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442681 2587 flags.go:64] FLAG: --v="2" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442685 2587 flags.go:64] FLAG: --version="false" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442689 2587 flags.go:64] FLAG: --vmodule="" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442692 2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.442696 2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442811 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.571429 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442815 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442819 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442822 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442824 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442827 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442830 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442835 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442838 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442841 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442843 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442846 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442849 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442851 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442854 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442856 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442859 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442861 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442864 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442867 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442869 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.572298 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442871 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442874 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442876 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442881 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442883 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442886 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442889 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442893 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442896 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442899 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442902 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442905 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442908 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442910 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442913 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442916 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442919 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442922 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442925 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442928 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.572918 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442930 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442933 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442936 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442939 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442941 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442944 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442947 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442949 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442952 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442954 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442957 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442959 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442962 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442964 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442967 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442971 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442974 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442976 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442996 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.442999 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.573511 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443002 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443005 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443007 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443010 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443013 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443016 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443018 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443021 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443023 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443026 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443030 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443033 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443035 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443038 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443040 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443043 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443046 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443049 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443051 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443054 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.574420 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443056 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443060 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443063 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443066 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443068 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443071 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443075 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443081 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443084 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443087 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443090 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443093 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443095 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443098 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443101 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443104 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443106 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.575141 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.443109 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.443653 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.459480 2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.459502 2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.459541 2587 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.459547 2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459626 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459632 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459636 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459639 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459642 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459645 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459648 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459651 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459654 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.575949 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459659 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459663 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459666 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459669 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459672 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459676 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459680 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459683 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459686 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459690 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459693 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459696 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459699 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459701 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459704 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459707 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459710 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459713 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459716 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459719 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.576403 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459722 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459725 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459727 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459731 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459734 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459736 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459739 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459742 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459744 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459747 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459750 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459752 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459755 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459758 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459760 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459763 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459766 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459769 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459772 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.579437 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459774 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459777 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459780 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459783 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459786 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459789 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459792 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459794 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459796 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459799 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459801 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459804 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459807 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459809 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459811 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459814 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459817 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459821 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459824 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459827 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.579992 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459830 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459833 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459839 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459842 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459844 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459847 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459849 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459852 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459855 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459857 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459862 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459865 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459868 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459870 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459873 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459876 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459878 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459881 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459884 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.580593 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459886 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459889 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459891 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459909 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459912 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459916 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459918 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459921 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459924 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459927 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459929 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.459932 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.459937 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460074 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460080 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:09:53.581224 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460088 2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460091 2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460094 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460097 2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460100 2587 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460103 2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460105 2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460108 2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460110 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460114 2587 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460117 2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460120 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460123 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460126 2587 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460128 2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460131 2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460134 2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460136 2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460139 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460141 2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:09:53.581835 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460144 2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460147 2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460149 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460152 2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460154 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460158 2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460162 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460165 2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460168 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460171 2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460174 2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460176 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460180 2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460182 2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460185 2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460188 2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460190 2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460193 2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460195 2587 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:09:53.583276 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460197 2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460200 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460204 2587 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460207 2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460210 2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460212 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460215 2587 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460217 2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460220 2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460223 2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460226 2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460230 2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460233 2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460236 2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460239 2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460241 2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460244 2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460246 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460249 2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:09:53.583842 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460251 2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460253 2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460256 2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460259 2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460261 2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460264 2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460269 2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460271 2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460274 2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460276 2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460278 2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460281 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460283 2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460286 2587 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460289 2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460294 2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460298 2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460301 2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460304 2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460307 2587 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:09:53.687371 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460310 2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460312 2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460315 2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460318 2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460320 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460323 2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460325 2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460328 2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460330 2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460333 2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460335 2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460338 2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460340 2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460342 2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460345 2587 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460348 2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460350 2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460353 2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:09:53.713176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:53.460356 2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.460362 2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.460501 2587 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.463282 2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.464514 2587 server.go:1019] "Starting client certificate rotation" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.467063 2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.469624 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.469661 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.472352 2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.485322 2587 log.go:25] "Validated CRI v1 runtime API" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.487761 2587 log.go:25] "Validated CRI v1 image API" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.488685 2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 18:09:53.714754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.491374 2587 fs.go:135] Filesystem UUIDs: map[2003a3c2-1841-4f9d-9c53-19d0fd0c798d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc2256fc-be64-4f6a-8b1e-57959db9e45d:/dev/nvme0n1p3] Jul 17 18:09:53.715357 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.491395 2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.496959 2587 manager.go:216] Machine: {Timestamp:2026-07-17 18:09:53.49643158 +0000 UTC m=+0.315202842 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103982 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d9a9c4a1a1c3897152b85a04793ec SystemUUID:ec2d9a9c-4a1a-1c38-9715-2b85a04793ec BootID:4bdaae38-c00e-4a7c-95e9-81fbca3faacb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:f2:cc:26:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:f2:cc:26:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:ae:62:93:f1:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.497084 2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.497213 2587 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.498274 2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.498298 2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.498968 2587 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.498976 2587 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 18:09:53.715572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.498996 2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.499008 2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.500320 2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.502469 2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504346 2587 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504456 2587 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504486 2587 kubelet.go:486] "Attempting to sync node with API server" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504496 2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504507 2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504516 2587 kubelet.go:398] "Adding apiserver pod source" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504528 2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504837 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.504851 2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.506666 2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.508653 2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.508681 2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509747 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509761 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509766 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509772 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509780 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509787 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509795 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509800 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509807 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509813 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509821 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509826 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509851 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.509856 2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.513410 2587 server.go:1268] "Started kubelet" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.513808 2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.513885 2587 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.514303 2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.515494 2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.516913 2587 server.go:310] "Adding debug handlers to kubelet server" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.517210 2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.517309 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.517406 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 18:09:53.716025 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.520942 2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.521941 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-225.ec2.internal.18c3261f5ce7b0d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-225.ec2.internal,UID:ip-10-0-136-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-225.ec2.internal,},FirstTimestamp:2026-07-17 18:09:53.513378004 +0000 UTC m=+0.332149259,LastTimestamp:2026-07-17 18:09:53.513378004 +0000 UTC m=+0.332149259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-225.ec2.internal,}" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.522950 2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.522970 2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.522950 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qlc8h" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.525237 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.525884 2587 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.525896 2587 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.525946 2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526024 2587 reconstruct.go:97] "Volume reconstruction finished" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526035 2587 reconciler.go:29] "Reconciler: start to sync state" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526402 2587 factory.go:55] Registering systemd factory Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526418 2587 factory.go:223] Registration of the systemd container factory successfully Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526660 2587 factory.go:153] Registering CRI-O factory Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526669 2587 factory.go:223] Registration of the crio container factory successfully Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526720 2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526744 2587 factory.go:103] Registering Raw factory Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.526763 2587 manager.go:1196] Started watching for new ooms in manager Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.527148 2587 manager.go:349] Starting recovery of all containers Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.528121 2587 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.528247 2587 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.529954 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qlc8h" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.544851 2587 manager.go:354] Recovery completed Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.554029 2587 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.560487 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.562566 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.562594 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.562606 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.569085 2587 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.569099 2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.569103 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.717369 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.569119 2587 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.570011 2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-225.ec2.internal.18c3261f5fd68180 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-225.ec2.internal,UID:ip-10-0-136-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-225.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-225.ec2.internal,},FirstTimestamp:2026-07-17 18:09:53.562583424 +0000 UTC m=+0.381354679,LastTimestamp:2026-07-17 18:09:53.562583424 +0000 UTC m=+0.381354679,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-225.ec2.internal,}" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.570076 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.570098 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.570107 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.571349 2587 policy_none.go:49] "None policy: Start" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.571366 2587 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.571381 2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.573522 2587 policy_none.go:47] "Start" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.625667 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.625796 2587 manager.go:337] "Starting Device Plugin manager" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.625962 2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.625976 2587 server.go:87] "Starting device plugin registration server" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.626306 2587 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.626319 2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.626648 2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.626716 2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.626724 2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.627325 2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 18:09:53.719394 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.627379 2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.726785 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.728444 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.728474 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.728488 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.728522 2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.731932 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.733229 2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.733259 2587 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.733278 2587 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.733315 2587 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.737291 2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.737312 2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-225.ec2.internal\": node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.737693 2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.788412 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.834025 2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal"] Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.834108 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.836171 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.836194 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.836205 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.838806 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.838972 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839014 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839584 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839610 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839621 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839625 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839644 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.839653 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.842643 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.842672 2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.843470 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.843496 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:09:53.848695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.843507 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.889185 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.891194 2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-225.ec2.internal\" not found" node="ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.905439 2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-225.ec2.internal\" not found" node="ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.929529 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.929565 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:53.981460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:53.929583 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c34073dec3e096eb00af0ee236229ab6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-225.ec2.internal\" (UID: \"c34073dec3e096eb00af0ee236229ab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:53.990183 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030745 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030779 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030798 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c34073dec3e096eb00af0ee236229ab6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-225.ec2.internal\" (UID: \"c34073dec3e096eb00af0ee236229ab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030852 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c34073dec3e096eb00af0ee236229ab6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-225.ec2.internal\" (UID: \"c34073dec3e096eb00af0ee236229ab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030858 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.030868 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/350c3421dacc1ed89bdb4106664b6923-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal\" (UID: \"350c3421dacc1ed89bdb4106664b6923\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.093918 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.090385 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:54.259867 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.194748 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:54.259867 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.194755 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.259867 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.209492 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.295114 2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-225.ec2.internal\" not found" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.324240 2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.326392 2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.338469 2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.340208 2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" Jul 17 18:09:54.352764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.350401 2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:09:54.460697 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.460672 2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:54.463863 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.463850 2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 18:09:54.463973 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.463954 2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:09:54.464049 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.464023 2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:09:54.464110 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.464023 2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:09:54.506443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.506411 2587 apiserver.go:52] "Watching apiserver" Jul 17 18:09:54.523246 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.523216 2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 18:09:54.533465 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.533442 2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 18:09:54.533814 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.533792 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-99kxg","openshift-image-registry/node-ca-lqzqv","openshift-network-operator/iptables-alerter-gp54z","openshift-ovn-kubernetes/ovnkube-node-ljz5r","kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2","openshift-dns/node-resolver-csmjl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal","openshift-multus/multus-2n6k9","openshift-multus/multus-additional-cni-plugins-xdn5t","openshift-multus/network-metrics-daemon-q6mz4","openshift-network-diagnostics/network-check-target-9lcq9","kube-system/global-pull-secret-syncer-4nfq7","kube-system/konnectivity-agent-v5fsc"] Jul 17 18:09:54.534305 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.534284 2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:09:54.536614 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.536597 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.538483 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.538467 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m4jrg\"" Jul 17 18:09:54.538811 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.538797 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.538851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.538826 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.539597 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.539572 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.541836 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.541816 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztbhl\"" Jul 17 18:09:54.541931 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.541841 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.541931 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.541902 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.542132 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.541889 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 18:09:54.542349 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.542321 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.542428 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.542351 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:04:54 +0000 UTC" deadline="2028-04-11 12:32:35.682710456 +0000 UTC" Jul 17 18:09:54.542428 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.542377 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15210h22m41.14033776s" Jul 17 18:09:54.544976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.544951 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8qgnd\"" Jul 17 18:09:54.545093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.544951 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.545093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.544997 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.545093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.545026 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 18:09:54.545370 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.545355 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.547480 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547456 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 18:09:54.547586 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547486 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.547586 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547467 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.547882 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547856 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 18:09:54.547965 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547894 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dlx5g\"" Jul 17 18:09:54.547965 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547930 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 18:09:54.547965 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547956 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.548399 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.547964 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 18:09:54.550155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.550135 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.550244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.550145 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.550480 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.550463 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lxtgc\"" Jul 17 18:09:54.550554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.550492 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 18:09:54.550607 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.550589 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.552753 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.552725 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.552870 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.552756 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jn8lw\"" Jul 17 18:09:54.552870 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.552725 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.553074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.553058 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.555300 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555284 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lld4k\"" Jul 17 18:09:54.555359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555334 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 18:09:54.555411 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555379 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 18:09:54.555573 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555557 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 18:09:54.555650 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555563 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 18:09:54.555793 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.555777 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.557447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.557427 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 18:09:54.557760 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.557745 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 18:09:54.557961 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.557945 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g2rkf\"" Jul 17 18:09:54.558000 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.557987 2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fck7c" Jul 17 18:09:54.558311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.558299 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.558381 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.558362 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:09:54.560778 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.560755 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:54.560861 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.560834 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:09:54.563246 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.563226 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.563336 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.563291 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:09:54.565747 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.565720 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.567550 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.567534 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 18:09:54.567787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.567773 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 18:09:54.567842 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.567805 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zx66f\"" Jul 17 18:09:54.567976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.567963 2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fck7c" Jul 17 18:09:54.613177 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.613155 2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:54.627390 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.627368 2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 18:09:54.635396 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635366 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-hostroot\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.635497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635398 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-sys\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.635497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635417 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28775f48-6fd5-429e-bf24-5567776b8861-serviceca\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.635497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635444 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.635497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635478 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635526 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-kubernetes\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635552 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-conf\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635589 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-conf-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635624 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635650 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrk7\" (UniqueName: \"kubernetes.io/projected/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-kube-api-access-kqrk7\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635672 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-socket-dir-parent\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.635729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635705 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-tuned\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635736 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-daemon-config\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635761 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysconfig\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635788 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-lib-modules\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635813 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-var-lib-kubelet\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635840 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf4g\" (UniqueName: \"kubernetes.io/projected/6e43ca7b-c520-4832-83e2-cba0828b253c-kube-api-access-4xf4g\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635869 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635895 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635919 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-script-lib\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635942 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/993fe589-95a1-4540-93a7-dec7b0e1a02f-host-slash\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635960 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-slash\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.635977 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636005 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e43c7842-165d-4505-b7a3-803c73c7c01d-hosts-file\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.636057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636020 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636033 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-host\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636066 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjg2\" (UniqueName: \"kubernetes.io/projected/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kube-api-access-8rjg2\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636084 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-cnibin\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636113 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43c7842-165d-4505-b7a3-803c73c7c01d-tmp-dir\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636138 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-etc-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636164 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-env-overrides\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636187 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-k8s-cni-cncf-io\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636202 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636232 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aad4f185-f146-4291-a953-ed2739ca5b82-konnectivity-ca\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636272 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-var-lib-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636303 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-ovn\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636331 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-log-socket\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636357 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-cnibin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.636583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636381 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzjs\" (UniqueName: \"kubernetes.io/projected/993fe589-95a1-4540-93a7-dec7b0e1a02f-kube-api-access-zfzjs\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636409 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-kubelet-config\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636428 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovn-node-metrics-cert\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636443 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-multus-certs\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636478 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-tmp\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636519 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636545 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-systemd-units\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636571 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-os-release\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636602 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-run\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636657 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-os-release\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636686 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw84l\" (UniqueName: \"kubernetes.io/projected/da86941c-593e-4b66-b353-52e1d08c9b99-kube-api-access-bw84l\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636711 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636738 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-bin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636763 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-modprobe-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.637244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636787 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-system-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636812 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-netns\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636837 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-node-log\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636862 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-netd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636887 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75sz\" (UniqueName: \"kubernetes.io/projected/28775f48-6fd5-429e-bf24-5567776b8861-kube-api-access-v75sz\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636915 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/993fe589-95a1-4540-93a7-dec7b0e1a02f-iptables-alerter-script\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636939 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb6t\" (UniqueName: \"kubernetes.io/projected/358e0c4e-84da-47c7-9c43-07c395d6d242-kube-api-access-lxb6t\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636964 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-systemd\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.636988 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-sys-fs\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637016 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-dbus\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637058 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-cni-binary-copy\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637093 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmbd\" (UniqueName: \"kubernetes.io/projected/02a40295-d82e-413d-b438-e0b99b4f5fdc-kube-api-access-kbmbd\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637118 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-device-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.637808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637157 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637186 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-systemd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637201 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-bin\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637218 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-config\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637233 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9ls\" (UniqueName: \"kubernetes.io/projected/e43c7842-165d-4505-b7a3-803c73c7c01d-kube-api-access-ch9ls\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637249 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-netns\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637263 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28775f48-6fd5-429e-bf24-5567776b8861-host\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637278 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-system-cni-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637295 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637311 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-kubelet\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637336 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-etc-kubernetes\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637353 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-socket-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637378 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-registration-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637393 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-multus\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.638217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637430 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-kubelet\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.638632 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637470 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.638632 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637498 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aad4f185-f146-4291-a953-ed2739ca5b82-agent-certs\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.638632 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.637534 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.737855 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737773 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-system-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.737855 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737811 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-netns\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.737855 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737829 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-node-log\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737871 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-netd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737885 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-node-log\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737901 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75sz\" (UniqueName: \"kubernetes.io/projected/28775f48-6fd5-429e-bf24-5567776b8861-kube-api-access-v75sz\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737929 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/993fe589-95a1-4540-93a7-dec7b0e1a02f-iptables-alerter-script\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737898 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-netns\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737932 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-netd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737950 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-system-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.737982 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb6t\" (UniqueName: \"kubernetes.io/projected/358e0c4e-84da-47c7-9c43-07c395d6d242-kube-api-access-lxb6t\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738032 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-systemd\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738058 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-sys-fs\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738083 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-dbus\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738107 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-cni-binary-copy\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738134 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmbd\" (UniqueName: \"kubernetes.io/projected/02a40295-d82e-413d-b438-e0b99b4f5fdc-kube-api-access-kbmbd\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738134 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-systemd\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.738217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738158 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-device-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738157 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-sys-fs\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738374 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-dbus\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738406 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-device-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738409 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738447 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-systemd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738474 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-bin\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738500 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-config\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738528 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9ls\" (UniqueName: \"kubernetes.io/projected/e43c7842-165d-4505-b7a3-803c73c7c01d-kube-api-access-ch9ls\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738534 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-systemd\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738553 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-netns\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738556 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-cni-bin\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738599 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-netns\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738639 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28775f48-6fd5-429e-bf24-5567776b8861-host\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.739074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738662 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/993fe589-95a1-4540-93a7-dec7b0e1a02f-iptables-alerter-script\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738674 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-system-cni-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738713 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-cni-binary-copy\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738725 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738732 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28775f48-6fd5-429e-bf24-5567776b8861-host\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738759 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-system-cni-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738771 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-kubelet\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738823 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-etc-kubernetes\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738829 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-kubelet\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738849 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-socket-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738886 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-registration-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738891 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-etc-kubernetes\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738913 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-multus\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738938 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-kubelet\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.739891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738949 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-registration-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738964 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.738996 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739008 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aad4f185-f146-4291-a953-ed2739ca5b82-agent-certs\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739010 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-kubelet\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739019 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-socket-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739011 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-multus\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739070 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739070 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739132 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739168 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-hostroot\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739187 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-config\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739206 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-hostroot\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739193 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-sys\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739223 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-sys\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.740660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739246 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28775f48-6fd5-429e-bf24-5567776b8861-serviceca\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739272 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739301 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739315 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739328 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-kubernetes\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739364 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-conf\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739383 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-kubernetes\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739404 2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739410 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-conf-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739461 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-conf-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739464 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739492 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-conf\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739501 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrk7\" (UniqueName: \"kubernetes.io/projected/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-kube-api-access-kqrk7\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.739530 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739549 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-cni-dir\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.741512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739583 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-socket-dir-parent\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739531 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-socket-dir-parent\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739625 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28775f48-6fd5-429e-bf24-5567776b8861-serviceca\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.739635 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.239586739 +0000 UTC m=+2.058358001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739685 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-tuned\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739711 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-daemon-config\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739737 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysconfig\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739761 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-lib-modules\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739762 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da86941c-593e-4b66-b353-52e1d08c9b99-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739832 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysconfig\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739861 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-var-lib-kubelet\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739880 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf4g\" (UniqueName: \"kubernetes.io/projected/6e43ca7b-c520-4832-83e2-cba0828b253c-kube-api-access-4xf4g\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739902 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739915 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-lib-modules\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.739920 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-var-lib-kubelet\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740028 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740053 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-script-lib\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740074 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/993fe589-95a1-4540-93a7-dec7b0e1a02f-host-slash\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740098 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-slash\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740121 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740149 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/993fe589-95a1-4540-93a7-dec7b0e1a02f-host-slash\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740162 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740167 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-slash\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740087 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740254 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e43c7842-165d-4505-b7a3-803c73c7c01d-hosts-file\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740280 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740288 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/358e0c4e-84da-47c7-9c43-07c395d6d242-multus-daemon-config\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740298 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-host\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.742707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740340 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjg2\" (UniqueName: \"kubernetes.io/projected/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kube-api-access-8rjg2\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740362 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e43c7842-165d-4505-b7a3-803c73c7c01d-hosts-file\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740344 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-host\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740379 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-cnibin\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740417 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43c7842-165d-4505-b7a3-803c73c7c01d-tmp-dir\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740430 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-sysctl-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740443 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-etc-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740469 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-env-overrides\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740509 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-cnibin\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740558 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-etc-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740557 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-k8s-cni-cncf-io\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740597 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-k8s-cni-cncf-io\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740597 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740635 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aad4f185-f146-4291-a953-ed2739ca5b82-konnectivity-ca\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740658 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-var-lib-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.743585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740679 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40a1a174-05d2-4dea-bc7a-0b7760b946c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740687 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-ovn\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740714 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-log-socket\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740740 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-cnibin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740750 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43c7842-165d-4505-b7a3-803c73c7c01d-tmp-dir\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740770 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzjs\" (UniqueName: \"kubernetes.io/projected/993fe589-95a1-4540-93a7-dec7b0e1a02f-kube-api-access-zfzjs\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740797 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-kubelet-config\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740826 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovn-node-metrics-cert\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740832 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-log-socket\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740853 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-multus-certs\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740861 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovnkube-script-lib\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740878 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-tmp\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740881 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-var-lib-openvswitch\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740914 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740923 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-env-overrides\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.744431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740937 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-run-ovn\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740944 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-systemd-units\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740971 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-os-release\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740994 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-cnibin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.740996 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-run\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741059 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-run\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741036 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-os-release\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741143 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw84l\" (UniqueName: \"kubernetes.io/projected/da86941c-593e-4b66-b353-52e1d08c9b99-kube-api-access-bw84l\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741177 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741212 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-bin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741218 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741235 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da86941c-593e-4b66-b353-52e1d08c9b99-os-release\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741239 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-modprobe-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741284 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-os-release\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741326 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aad4f185-f146-4291-a953-ed2739ca5b82-konnectivity-ca\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.745252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741147 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-run-multus-certs\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741356 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-modprobe-d\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741181 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-systemd-units\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741412 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73cf7878-9139-48a9-a8ea-5c958c28dd3d-kubelet-config\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.741418 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/358e0c4e-84da-47c7-9c43-07c395d6d242-host-var-lib-cni-bin\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.741476 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.741551 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.241533506 +0000 UTC m=+2.060304751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.743069 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-etc-tuned\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.743261 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e43ca7b-c520-4832-83e2-cba0828b253c-tmp\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.743286 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aad4f185-f146-4291-a953-ed2739ca5b82-agent-certs\") pod \"konnectivity-agent-v5fsc\" (UID: \"aad4f185-f146-4291-a953-ed2739ca5b82\") " pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.743397 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-ovn-node-metrics-cert\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.746170 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.745925 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75sz\" (UniqueName: \"kubernetes.io/projected/28775f48-6fd5-429e-bf24-5567776b8861-kube-api-access-v75sz\") pod \"node-ca-lqzqv\" (UID: \"28775f48-6fd5-429e-bf24-5567776b8861\") " pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.746813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.746421 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9ls\" (UniqueName: \"kubernetes.io/projected/e43c7842-165d-4505-b7a3-803c73c7c01d-kube-api-access-ch9ls\") pod \"node-resolver-csmjl\" (UID: \"e43c7842-165d-4505-b7a3-803c73c7c01d\") " pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.746813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.746725 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb6t\" (UniqueName: \"kubernetes.io/projected/358e0c4e-84da-47c7-9c43-07c395d6d242-kube-api-access-lxb6t\") pod \"multus-2n6k9\" (UID: \"358e0c4e-84da-47c7-9c43-07c395d6d242\") " pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.746813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.746734 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmbd\" (UniqueName: \"kubernetes.io/projected/02a40295-d82e-413d-b438-e0b99b4f5fdc-kube-api-access-kbmbd\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:54.749512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.749491 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrk7\" (UniqueName: \"kubernetes.io/projected/3fbdc11d-1db8-47f7-8675-f5c1b9ff6158-kube-api-access-kqrk7\") pod \"ovnkube-node-ljz5r\" (UID: \"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.750975 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.750957 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:54.751108 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.750978 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:54.751108 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.750992 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:54.751108 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:54.751045 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:55.251031401 +0000 UTC m=+2.069802645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:54.753274 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.753255 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf4g\" (UniqueName: \"kubernetes.io/projected/6e43ca7b-c520-4832-83e2-cba0828b253c-kube-api-access-4xf4g\") pod \"tuned-99kxg\" (UID: \"6e43ca7b-c520-4832-83e2-cba0828b253c\") " pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.754885 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.754853 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw84l\" (UniqueName: \"kubernetes.io/projected/da86941c-593e-4b66-b353-52e1d08c9b99-kube-api-access-bw84l\") pod \"multus-additional-cni-plugins-xdn5t\" (UID: \"da86941c-593e-4b66-b353-52e1d08c9b99\") " pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.755209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.755190 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzjs\" (UniqueName: \"kubernetes.io/projected/993fe589-95a1-4540-93a7-dec7b0e1a02f-kube-api-access-zfzjs\") pod \"iptables-alerter-gp54z\" (UID: \"993fe589-95a1-4540-93a7-dec7b0e1a02f\") " pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.755449 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.755429 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjg2\" (UniqueName: \"kubernetes.io/projected/40a1a174-05d2-4dea-bc7a-0b7760b946c1-kube-api-access-8rjg2\") pod \"aws-ebs-csi-driver-node-gcrv2\" (UID: \"40a1a174-05d2-4dea-bc7a-0b7760b946c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.864232 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.864190 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99kxg" Jul 17 18:09:54.869402 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.869380 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lqzqv" Jul 17 18:09:54.897346 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.897315 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gp54z" Jul 17 18:09:54.911541 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.911513 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:09:54.911855 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.911826 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993fe589_95a1_4540_93a7_dec7b0e1a02f.slice/crio-14a4532b7b860a333d50407b3407a90579a8d76b1f9e4fc236605211def198c4 WatchSource:0}: Error finding container 14a4532b7b860a333d50407b3407a90579a8d76b1f9e4fc236605211def198c4: Status 404 returned error can't find the container with id 14a4532b7b860a333d50407b3407a90579a8d76b1f9e4fc236605211def198c4 Jul 17 18:09:54.912377 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.912354 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34073dec3e096eb00af0ee236229ab6.slice/crio-e3ccccde60107545069c234207bbdd1c5627e0d02cd9238b686a64b6c7d72be0 WatchSource:0}: Error finding container e3ccccde60107545069c234207bbdd1c5627e0d02cd9238b686a64b6c7d72be0: Status 404 returned error can't find the container with id e3ccccde60107545069c234207bbdd1c5627e0d02cd9238b686a64b6c7d72be0 Jul 17 18:09:54.919664 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.919626 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:09:54.922909 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.922887 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbdc11d_1db8_47f7_8675_f5c1b9ff6158.slice/crio-fa18338014db3f42e4fcbe0a7c4dc13f545bd67a0556af01317094a788bb905b WatchSource:0}: Error finding container fa18338014db3f42e4fcbe0a7c4dc13f545bd67a0556af01317094a788bb905b: Status 404 returned error can't find the container with id fa18338014db3f42e4fcbe0a7c4dc13f545bd67a0556af01317094a788bb905b Jul 17 18:09:54.929951 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.929929 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" Jul 17 18:09:54.939111 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.939088 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a1a174_05d2_4dea_bc7a_0b7760b946c1.slice/crio-85254ef1b6f2aa3fe114f88c74aa73b5abb6395c965be388589e62d749900ce6 WatchSource:0}: Error finding container 85254ef1b6f2aa3fe114f88c74aa73b5abb6395c965be388589e62d749900ce6: Status 404 returned error can't find the container with id 85254ef1b6f2aa3fe114f88c74aa73b5abb6395c965be388589e62d749900ce6 Jul 17 18:09:54.942153 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.942131 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-csmjl" Jul 17 18:09:54.950688 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.950665 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43c7842_165d_4505_b7a3_803c73c7c01d.slice/crio-09ee887454715b9d71181c778e3b9cc714bf0b47abac8478a039e7ca4e2610c4 WatchSource:0}: Error finding container 09ee887454715b9d71181c778e3b9cc714bf0b47abac8478a039e7ca4e2610c4: Status 404 returned error can't find the container with id 09ee887454715b9d71181c778e3b9cc714bf0b47abac8478a039e7ca4e2610c4 Jul 17 18:09:54.952003 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.951985 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2n6k9" Jul 17 18:09:54.960562 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.960542 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e0c4e_84da_47c7_9c43_07c395d6d242.slice/crio-263b0b7a4c3664b265480254652018426a6aac494321dd029d3f137f1f978a5b WatchSource:0}: Error finding container 263b0b7a4c3664b265480254652018426a6aac494321dd029d3f137f1f978a5b: Status 404 returned error can't find the container with id 263b0b7a4c3664b265480254652018426a6aac494321dd029d3f137f1f978a5b Jul 17 18:09:54.966354 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.966332 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" Jul 17 18:09:54.971423 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:54.971397 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:09:54.974669 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.974648 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda86941c_593e_4b66_b353_52e1d08c9b99.slice/crio-1dd1023e2e87adc812154dedc87c9d96b8b2567c6332709f88f9ea7df7aa095f WatchSource:0}: Error finding container 1dd1023e2e87adc812154dedc87c9d96b8b2567c6332709f88f9ea7df7aa095f: Status 404 returned error can't find the container with id 1dd1023e2e87adc812154dedc87c9d96b8b2567c6332709f88f9ea7df7aa095f Jul 17 18:09:54.979963 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:54.979940 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad4f185_f146_4291_a953_ed2739ca5b82.slice/crio-581a412d17a695802d7215c0ee987880aa2055c50842edd4bdb62ed900ab753f WatchSource:0}: Error finding container 581a412d17a695802d7215c0ee987880aa2055c50842edd4bdb62ed900ab753f: Status 404 returned error can't find the container with id 581a412d17a695802d7215c0ee987880aa2055c50842edd4bdb62ed900ab753f Jul 17 18:09:55.154117 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:55.154084 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350c3421dacc1ed89bdb4106664b6923.slice/crio-1a1453568bcc4ea294092f15a56b11fb38ba3c763b2e1c100677442ac4db3f6a WatchSource:0}: Error finding container 1a1453568bcc4ea294092f15a56b11fb38ba3c763b2e1c100677442ac4db3f6a: Status 404 returned error can't find the container with id 1a1453568bcc4ea294092f15a56b11fb38ba3c763b2e1c100677442ac4db3f6a Jul 17 18:09:55.244526 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.244484 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:55.244705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.244543 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:55.244705 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.244648 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:55.244705 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.244652 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:55.244705 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.244703 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:09:56.244689646 +0000 UTC m=+3.063460888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:55.244850 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.244717 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:09:56.244710033 +0000 UTC m=+3.063481277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:55.318285 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:55.318245 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28775f48_6fd5_429e_bf24_5567776b8861.slice/crio-849b00de9e0e2f94cf5d77ffe182e081335d45e960738a8b633cb2e513b671f7 WatchSource:0}: Error finding container 849b00de9e0e2f94cf5d77ffe182e081335d45e960738a8b633cb2e513b671f7: Status 404 returned error can't find the container with id 849b00de9e0e2f94cf5d77ffe182e081335d45e960738a8b633cb2e513b671f7 Jul 17 18:09:55.328352 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:09:55.328329 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e43ca7b_c520_4832_83e2_cba0828b253c.slice/crio-55030223cb0f7799db7f4a36182928259a28cc61b99befd4ecf29443d0be1e6c WatchSource:0}: Error finding container 55030223cb0f7799db7f4a36182928259a28cc61b99befd4ecf29443d0be1e6c: Status 404 returned error can't find the container with id 55030223cb0f7799db7f4a36182928259a28cc61b99befd4ecf29443d0be1e6c Jul 17 18:09:55.345526 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.345479 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:55.345681 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.345629 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:55.345681 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.345655 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:55.345782 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.345670 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:55.345832 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.345781 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:56.345759932 +0000 UTC m=+3.164531180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:55.571126 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.570997 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:04:55 +0000 UTC" deadline="2028-03-24 17:46:31.52599066 +0000 UTC" Jul 17 18:09:55.571126 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.571068 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14783h36m35.95492738s" Jul 17 18:09:55.734896 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.734860 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:55.735107 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:55.735013 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:09:55.750286 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.750183 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2n6k9" event={"ID":"358e0c4e-84da-47c7-9c43-07c395d6d242","Type":"ContainerStarted","Data":"263b0b7a4c3664b265480254652018426a6aac494321dd029d3f137f1f978a5b"} Jul 17 18:09:55.756338 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.756301 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-csmjl" event={"ID":"e43c7842-165d-4505-b7a3-803c73c7c01d","Type":"ContainerStarted","Data":"09ee887454715b9d71181c778e3b9cc714bf0b47abac8478a039e7ca4e2610c4"} Jul 17 18:09:55.760761 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.760728 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" event={"ID":"40a1a174-05d2-4dea-bc7a-0b7760b946c1","Type":"ContainerStarted","Data":"85254ef1b6f2aa3fe114f88c74aa73b5abb6395c965be388589e62d749900ce6"} Jul 17 18:09:55.776090 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.772771 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gp54z" event={"ID":"993fe589-95a1-4540-93a7-dec7b0e1a02f","Type":"ContainerStarted","Data":"14a4532b7b860a333d50407b3407a90579a8d76b1f9e4fc236605211def198c4"} Jul 17 18:09:55.782198 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.782095 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" event={"ID":"c34073dec3e096eb00af0ee236229ab6","Type":"ContainerStarted","Data":"e3ccccde60107545069c234207bbdd1c5627e0d02cd9238b686a64b6c7d72be0"} Jul 17 18:09:55.797167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.797056 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lqzqv" event={"ID":"28775f48-6fd5-429e-bf24-5567776b8861","Type":"ContainerStarted","Data":"849b00de9e0e2f94cf5d77ffe182e081335d45e960738a8b633cb2e513b671f7"} Jul 17 18:09:55.818594 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.818554 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" event={"ID":"350c3421dacc1ed89bdb4106664b6923","Type":"ContainerStarted","Data":"1a1453568bcc4ea294092f15a56b11fb38ba3c763b2e1c100677442ac4db3f6a"} Jul 17 18:09:55.833328 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.833230 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5fsc" event={"ID":"aad4f185-f146-4291-a953-ed2739ca5b82","Type":"ContainerStarted","Data":"581a412d17a695802d7215c0ee987880aa2055c50842edd4bdb62ed900ab753f"} Jul 17 18:09:55.869267 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.868809 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"fa18338014db3f42e4fcbe0a7c4dc13f545bd67a0556af01317094a788bb905b"} Jul 17 18:09:55.882723 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.882645 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99kxg" event={"ID":"6e43ca7b-c520-4832-83e2-cba0828b253c","Type":"ContainerStarted","Data":"55030223cb0f7799db7f4a36182928259a28cc61b99befd4ecf29443d0be1e6c"} Jul 17 18:09:55.904599 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.904548 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerStarted","Data":"1dd1023e2e87adc812154dedc87c9d96b8b2567c6332709f88f9ea7df7aa095f"} Jul 17 18:09:55.989336 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:55.989302 2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:56.255098 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.254928 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:56.255282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.255108 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:56.255282 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.255273 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:56.255423 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.255343 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:09:58.255321753 +0000 UTC m=+5.074093000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:56.255423 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.255421 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:56.255541 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.255458 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:09:58.255446098 +0000 UTC m=+5.074217345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:56.344174 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.343921 2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:56.357009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.356374 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:56.357009 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.356552 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:56.357009 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.356576 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:56.357009 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.356590 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:56.357009 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.356649 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:09:58.356628933 +0000 UTC m=+5.175400181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:56.572172 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.571940 2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:04:55 +0000 UTC" deadline="2027-12-24 00:38:33.073907838 +0000 UTC" Jul 17 18:09:56.572172 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.572069 2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12582h28m36.501843392s" Jul 17 18:09:56.734600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.733804 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:56.734600 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.733961 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:09:56.734600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:56.734080 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:56.734600 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:56.734213 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:09:57.600094 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:57.600044 2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:09:57.736367 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:57.736308 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:57.736542 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:57.736459 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:09:58.273072 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:58.272957 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:58.273256 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:58.273084 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:58.273256 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.273229 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:58.273379 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.273296 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:10:02.273276785 +0000 UTC m=+9.092048035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:09:58.273739 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.273714 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:58.273838 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.273777 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:10:02.273760345 +0000 UTC m=+9.092531589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:09:58.374109 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:58.374067 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:58.374388 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.374281 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:09:58.374388 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.374308 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:09:58.374388 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.374324 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:58.374564 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.374392 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:02.374372182 +0000 UTC m=+9.193143431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:09:58.734438 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:58.734286 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:09:58.734939 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.734439 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:09:58.734939 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:58.734858 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:09:58.735091 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:58.734954 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:09:59.736573 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:09:59.736535 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:09:59.737050 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:09:59.736686 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:00.734473 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:00.734431 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:00.734473 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:00.734466 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:00.734779 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:00.734596 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:00.734779 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:00.734697 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:01.734091 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:01.734044 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:01.734549 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:01.734177 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:02.312051 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:02.312015 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:02.312252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:02.312100 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:02.312252 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.312239 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:02.312372 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.312305 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:10:10.312284595 +0000 UTC m=+17.131055843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:02.312372 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.312370 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:02.312491 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.312399 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:10:10.312387834 +0000 UTC m=+17.131159082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:02.413527 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:02.413401 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:02.413713 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.413564 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:10:02.413713 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.413584 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:10:02.413713 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.413595 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:02.413713 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.413662 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:10.41364423 +0000 UTC m=+17.232415474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:02.734395 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:02.734302 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:02.734395 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:02.734358 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:02.735066 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.734451 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:02.735066 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:02.734591 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:03.735790 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:03.735284 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:03.735790 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:03.735401 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:04.733662 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:04.733626 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:04.733662 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:04.733670 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:04.733911 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:04.733764 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:04.733971 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:04.733897 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:05.734575 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:05.734536 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:05.735084 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:05.734671 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:06.734329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:06.734288 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:06.734516 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:06.734288 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:06.734516 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:06.734429 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:06.734516 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:06.734487 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:07.733665 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:07.733571 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:07.733824 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:07.733724 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:08.734089 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:08.734050 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:08.734089 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:08.734090 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:08.734599 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:08.734185 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:08.734599 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:08.734333 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:09.733769 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:09.733728 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:09.733959 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:09.733868 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:10.371590 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:10.371550 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:10.372071 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:10.371608 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:10.372071 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.371708 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:10.372071 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.371785 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:10:26.371765095 +0000 UTC m=+33.190536337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:10.372071 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.371718 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:10.372071 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.371835 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:10:26.371823723 +0000 UTC m=+33.190594967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:10.472894 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:10.472860 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:10.473089 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.473052 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:10:10.473089 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.473080 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:10:10.473196 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.473093 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:10.473196 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.473152 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:26.473132387 +0000 UTC m=+33.291903633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:10.733585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:10.733466 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:10.733585 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:10.733489 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:10.733803 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.733598 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:10.733803 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:10.733697 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:11.734560 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:11.734519 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:11.735001 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:11.734673 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:12.734416 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:12.734199 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:12.734613 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:12.734237 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:12.734613 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:12.734490 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:12.734613 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:12.734596 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:13.734669 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:13.734633 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:13.735143 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:13.734740 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:14.734533 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:14.734488 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:14.734726 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:14.734495 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:14.734726 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:14.734610 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:14.734726 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:14.734712 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:15.733507 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:15.733488 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:15.733598 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:15.733580 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:15.979919 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:15.979883 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" event={"ID":"c34073dec3e096eb00af0ee236229ab6","Type":"ContainerStarted","Data":"69515a14cfc2565bb5c5f19006b73f00a886f8196aa0d7128f0c7750b6ed165f"} Jul 17 18:10:15.996100 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:15.995438 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-225.ec2.internal" podStartSLOduration=21.995418348 podStartE2EDuration="21.995418348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:15.995098919 +0000 UTC m=+22.813870185" watchObservedRunningTime="2026-07-17 18:10:15.995418348 +0000 UTC m=+22.814189623" Jul 17 18:10:16.733738 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.733488 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:16.733910 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:16.733856 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:16.733972 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.733578 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:16.734199 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:16.734142 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:16.983747 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.983663 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99kxg" event={"ID":"6e43ca7b-c520-4832-83e2-cba0828b253c","Type":"ContainerStarted","Data":"6462589029f02517dc2d4ec58ff5c428d8d1227d913173d783f8638c03061f9e"} Jul 17 18:10:16.985250 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.985227 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="1e8ec4dff3de8b6ac2e17daa7930fe6dd377802bb07f6207ac57328719692ba9" exitCode=0 Jul 17 18:10:16.985347 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.985299 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"1e8ec4dff3de8b6ac2e17daa7930fe6dd377802bb07f6207ac57328719692ba9"} Jul 17 18:10:16.986907 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.986875 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2n6k9" event={"ID":"358e0c4e-84da-47c7-9c43-07c395d6d242","Type":"ContainerStarted","Data":"d3df0768638fb7baea998e6aab7dd44fdf20e9b86781fce903309dcf858af638"} Jul 17 18:10:16.988387 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.988355 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-csmjl" event={"ID":"e43c7842-165d-4505-b7a3-803c73c7c01d","Type":"ContainerStarted","Data":"04447a6b82b8f11ab324ceaa20ee0f15aa7b090bebfc76feb6d244c0478e190e"} Jul 17 18:10:16.989785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.989766 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" event={"ID":"40a1a174-05d2-4dea-bc7a-0b7760b946c1","Type":"ContainerStarted","Data":"05c503223aaab4955db44b103a23ffdeb6a96877a7843c13d892d70a14f359bd"} Jul 17 18:10:16.993045 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.993023 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lqzqv" event={"ID":"28775f48-6fd5-429e-bf24-5567776b8861","Type":"ContainerStarted","Data":"99156e214f3a3bcd12d0bfb1002bae198cf5fffeb977c3043856b13374c4f29c"} Jul 17 18:10:16.994350 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.994332 2587 generic.go:356] "Generic (PLEG): container finished" podID="350c3421dacc1ed89bdb4106664b6923" containerID="4470d595fe3f927d1fa1c8f277da28d0f7ef4ed84b801967631f1f5e4b48957c" exitCode=0 Jul 17 18:10:16.994436 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.994406 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" event={"ID":"350c3421dacc1ed89bdb4106664b6923","Type":"ContainerDied","Data":"4470d595fe3f927d1fa1c8f277da28d0f7ef4ed84b801967631f1f5e4b48957c"} Jul 17 18:10:16.995750 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.995726 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5fsc" event={"ID":"aad4f185-f146-4291-a953-ed2739ca5b82","Type":"ContainerStarted","Data":"81ffd5315eb63579f1f2ad3d4c50a100cf4a22aa46917bac1bc70fe93c614a9f"} Jul 17 18:10:16.998293 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998277 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:10:16.998581 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998561 2587 generic.go:356] "Generic (PLEG): container finished" podID="3fbdc11d-1db8-47f7-8675-f5c1b9ff6158" containerID="289a8677c6838782396dbde468a4890e1aac18fc73414e6446a4851776e86239" exitCode=1 Jul 17 18:10:16.998655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998638 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"6a576dc1d6b831f9c56bf132620f31fe3e7b2ba39137b6ea582eda28d971a686"} Jul 17 18:10:16.998706 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998695 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"92e66314858720528f7b5f700aaec90618eb9b562d85f216482afe9e882a148c"} Jul 17 18:10:16.998737 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998728 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"7f32ddd7608e45c8a37bee487da4e39b1c9dc38594fe7a76991f6e0aa4120481"} Jul 17 18:10:16.998768 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998756 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"beb190c73cb8938a957f9797df0dc1d24c836874ca1039786a5fa373034751eb"} Jul 17 18:10:16.998804 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998786 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerDied","Data":"289a8677c6838782396dbde468a4890e1aac18fc73414e6446a4851776e86239"} Jul 17 18:10:16.998835 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:16.998815 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"9e0127cc4b16829f3f866f15ddbeb35802772291d3c56cef1af35a1b28a25f2b"} Jul 17 18:10:17.001341 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.001304 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-99kxg" podStartSLOduration=2.578832766 podStartE2EDuration="23.00129201s" totalImagesPullingTime="20.422459244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:17.000615293 +0000 UTC m=+23.819386585" watchObservedRunningTime="2026-07-17 18:10:17.00129201 +0000 UTC m=+23.820063275" Jul 17 18:10:17.035049 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.034832 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-csmjl" podStartSLOduration=2.237572097 podStartE2EDuration="23.034813658s" totalImagesPullingTime="20.797241561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:17.016000047 +0000 UTC m=+23.834771308" watchObservedRunningTime="2026-07-17 18:10:17.034813658 +0000 UTC m=+23.853584922" Jul 17 18:10:17.035167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.035120 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2n6k9" podStartSLOduration=2.230530783 podStartE2EDuration="23.035112849s" totalImagesPullingTime="20.804582066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:17.033839829 +0000 UTC m=+23.852611092" watchObservedRunningTime="2026-07-17 18:10:17.035112849 +0000 UTC m=+23.853884113" Jul 17 18:10:17.091524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.091479 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v5fsc" podStartSLOduration=2.356486648 podStartE2EDuration="23.091462202s" totalImagesPullingTime="20.734975554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:17.091463881 +0000 UTC m=+23.910235144" watchObservedRunningTime="2026-07-17 18:10:17.091462202 +0000 UTC m=+23.910233460" Jul 17 18:10:17.107883 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.107838 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lqzqv" podStartSLOduration=2.710888606 podStartE2EDuration="23.107823209s" totalImagesPullingTime="20.396934603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:17.107747325 +0000 UTC m=+23.926518589" watchObservedRunningTime="2026-07-17 18:10:17.107823209 +0000 UTC m=+23.926594473" Jul 17 18:10:17.734228 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:17.734189 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:17.734506 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:17.734315 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:18.004089 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.004052 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gp54z" event={"ID":"993fe589-95a1-4540-93a7-dec7b0e1a02f","Type":"ContainerStarted","Data":"a64c7d3bd84263486dff6cf8ea942a32e22c7fe62dd2693e27fb348a9d225bcb"} Jul 17 18:10:18.006517 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.006480 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" event={"ID":"350c3421dacc1ed89bdb4106664b6923","Type":"ContainerStarted","Data":"fc55de39d70276332a4ba63106d16cda1d14d8aeaef8594deaf87f849fd815ea"} Jul 17 18:10:18.028505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.028449 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gp54z" podStartSLOduration=3.215405083 podStartE2EDuration="24.028429221s" totalImagesPullingTime="20.813024138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:18.028301172 +0000 UTC m=+24.847072438" watchObservedRunningTime="2026-07-17 18:10:18.028429221 +0000 UTC m=+24.847200480" Jul 17 18:10:18.042167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.042117 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-225.ec2.internal" podStartSLOduration=24.042100656 podStartE2EDuration="24.042100656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:18.041420192 +0000 UTC m=+24.860191460" watchObservedRunningTime="2026-07-17 18:10:18.042100656 +0000 UTC m=+24.860871922" Jul 17 18:10:18.108616 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.108450 2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 18:10:18.319586 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.319551 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:10:18.320362 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.320343 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:10:18.640073 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.639899 2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:10:18.10846408Z","UUID":"1a2c6d24-9cd6-477a-8159-58d2e16b1c6a","Handler":null,"Name":"","Endpoint":""} Jul 17 18:10:18.641613 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.641591 2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 18:10:18.641754 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.641630 2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 18:10:18.733528 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.733454 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:18.733528 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:18.733496 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:18.733720 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:18.733567 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:18.733720 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:18.733693 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:19.010475 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:19.010442 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" event={"ID":"40a1a174-05d2-4dea-bc7a-0b7760b946c1","Type":"ContainerStarted","Data":"0e3dbdafbafb95289b062843674707ee29bf09d78e7e23d9b9407aef8a83551a"} Jul 17 18:10:19.010892 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:19.010869 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:10:19.011321 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:19.011304 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v5fsc" Jul 17 18:10:19.734252 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:19.734172 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:19.734408 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:19.734290 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:20.015954 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.015925 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:10:20.016468 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.016398 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"317d43a08872d6a64e079c6d792d73048e2108665733eedec386a9e1f2693288"} Jul 17 18:10:20.018531 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.018493 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" event={"ID":"40a1a174-05d2-4dea-bc7a-0b7760b946c1","Type":"ContainerStarted","Data":"92a6b54cb75965d6b6ca84a44c6ca37aa847e0a43307b3ebd50d67369b57d7c5"} Jul 17 18:10:20.038264 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.038214 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcrv2" podStartSLOduration=2.918669158 podStartE2EDuration="26.038197649s" totalImagesPullingTime="23.119528491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:20.037904875 +0000 UTC m=+26.856676139" watchObservedRunningTime="2026-07-17 18:10:20.038197649 +0000 UTC m=+26.856968912" Jul 17 18:10:20.733497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.733454 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:20.733684 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:20.733462 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:20.733684 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:20.733583 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:20.733684 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:20.733650 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:21.733532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:21.733460 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:21.734251 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:21.733557 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:22.024603 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.024571 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="18dea47416b6c97044fe05e6a212b59167bc03dbd84a050678189f357f8a9843" exitCode=0 Jul 17 18:10:22.024802 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.024662 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"18dea47416b6c97044fe05e6a212b59167bc03dbd84a050678189f357f8a9843"} Jul 17 18:10:22.028147 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028127 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:10:22.028512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028493 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"4d26a16624d9992af193e4f55d42efe9b19ce09288f76354d8bb10f23f229cb0"} Jul 17 18:10:22.028787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028765 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:22.028889 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028827 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:22.028889 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028878 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:22.028999 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.028925 2587 scope.go:117] "RemoveContainer" containerID="289a8677c6838782396dbde468a4890e1aac18fc73414e6446a4851776e86239" Jul 17 18:10:22.044669 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.044651 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:22.045596 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.045582 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:22.733696 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.733496 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:22.734248 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:22.733500 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:22.734248 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:22.733793 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:22.734248 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:22.733899 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:23.034554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.034534 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:10:23.034875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.034855 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" event={"ID":"3fbdc11d-1db8-47f7-8675-f5c1b9ff6158","Type":"ContainerStarted","Data":"9ac76e5d188b538700efe57eddb804237877e84a132dfc2cb89c82e202ec61f8"} Jul 17 18:10:23.066999 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.066921 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" podStartSLOduration=7.954014603 podStartE2EDuration="29.066900972s" totalImagesPullingTime="21.112886369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:23.066665145 +0000 UTC m=+29.885436412" watchObservedRunningTime="2026-07-17 18:10:23.066900972 +0000 UTC m=+29.885672239" Jul 17 18:10:23.376009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.375909 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9lcq9"] Jul 17 18:10:23.376197 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.376031 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:23.376197 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:23.376110 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:23.383380 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.383354 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4nfq7"] Jul 17 18:10:23.383524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.383455 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:23.383579 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:23.383550 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:23.384080 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.384053 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q6mz4"] Jul 17 18:10:23.384183 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:23.384158 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:23.384289 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:23.384267 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:24.038951 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:24.038908 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="325e0e3008b8ddda041452e37b6a16fc51724b94bdb27dc93204bc2a42e5983d" exitCode=0 Jul 17 18:10:24.039479 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:24.039005 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"325e0e3008b8ddda041452e37b6a16fc51724b94bdb27dc93204bc2a42e5983d"} Jul 17 18:10:24.734144 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:24.734098 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:24.734144 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:24.734142 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:24.734395 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:24.734208 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:24.734395 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:24.734339 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:25.737035 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:25.736841 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:25.737399 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:25.737123 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:26.046663 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.046625 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="a7c87ca23db3b504b723ca327f7272427a32267019078aa77c5e45c65720b228" exitCode=0 Jul 17 18:10:26.046830 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.046682 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"a7c87ca23db3b504b723ca327f7272427a32267019078aa77c5e45c65720b228"} Jul 17 18:10:26.390443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.390358 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:26.390443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.390414 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:26.390624 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.390500 2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:26.390624 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.390501 2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:26.390624 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.390559 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret podName:73cf7878-9139-48a9-a8ea-5c958c28dd3d nodeName:}" failed. No retries permitted until 2026-07-17 18:10:58.39054497 +0000 UTC m=+65.209316217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret") pod "global-pull-secret-syncer-4nfq7" (UID: "73cf7878-9139-48a9-a8ea-5c958c28dd3d") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:10:26.390624 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.390574 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs podName:02a40295-d82e-413d-b438-e0b99b4f5fdc nodeName:}" failed. No retries permitted until 2026-07-17 18:10:58.390567957 +0000 UTC m=+65.209339198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs") pod "network-metrics-daemon-q6mz4" (UID: "02a40295-d82e-413d-b438-e0b99b4f5fdc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:10:26.490794 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.490756 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:26.490969 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.490945 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:10:26.491041 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.490994 2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:10:26.491041 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.491009 2587 projected.go:196] Error preparing data for projected volume kube-api-access-6h28c for pod openshift-network-diagnostics/network-check-target-9lcq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:26.491141 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.491082 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c podName:9150e77e-d628-426c-8b6c-791aee78fd73 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:58.49106205 +0000 UTC m=+65.309833301 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6h28c" (UniqueName: "kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c") pod "network-check-target-9lcq9" (UID: "9150e77e-d628-426c-8b6c-791aee78fd73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:10:26.734212 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.734128 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:26.734212 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:26.734165 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:26.734403 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.734264 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:26.734469 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:26.734392 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:27.734507 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:27.734424 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:27.734972 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:27.734572 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6mz4" podUID="02a40295-d82e-413d-b438-e0b99b4f5fdc" Jul 17 18:10:28.734202 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:28.734159 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:28.734396 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:28.734180 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:28.734396 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:28.734294 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4nfq7" podUID="73cf7878-9139-48a9-a8ea-5c958c28dd3d" Jul 17 18:10:28.734396 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:28.734359 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9lcq9" podUID="9150e77e-d628-426c-8b6c-791aee78fd73" Jul 17 18:10:28.946964 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:28.946882 2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-225.ec2.internal" event="NodeReady" Jul 17 18:10:28.947357 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:28.947046 2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 18:10:28.987451 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:28.987418 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8w8gb"] Jul 17 18:10:29.008730 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.008695 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z46x2"] Jul 17 18:10:29.008924 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.008807 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.011160 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.011103 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 18:10:29.011160 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.011122 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 18:10:29.011409 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.011176 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkv2q\"" Jul 17 18:10:29.022863 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.022830 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8w8gb"] Jul 17 18:10:29.023052 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.022884 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z46x2"] Jul 17 18:10:29.023052 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.023002 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.025957 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.025593 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 18:10:29.025957 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.025773 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 18:10:29.025957 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.025775 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 18:10:29.025957 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.025838 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7247w\"" Jul 17 18:10:29.109195 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109157 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.109378 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109225 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bff42af5-1a78-43de-a2c6-2dabf7412bd7-cert\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.109378 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109255 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5wk\" (UniqueName: \"kubernetes.io/projected/bff42af5-1a78-43de-a2c6-2dabf7412bd7-kube-api-access-ft5wk\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.109378 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109359 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9377ef-9af1-4c91-a65e-c831b6874962-config-volume\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.109503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109414 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9377ef-9af1-4c91-a65e-c831b6874962-tmp-dir\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.109503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.109451 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdkw\" (UniqueName: \"kubernetes.io/projected/dc9377ef-9af1-4c91-a65e-c831b6874962-kube-api-access-kjdkw\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.210552 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210464 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bff42af5-1a78-43de-a2c6-2dabf7412bd7-cert\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.210552 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210515 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5wk\" (UniqueName: \"kubernetes.io/projected/bff42af5-1a78-43de-a2c6-2dabf7412bd7-kube-api-access-ft5wk\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.210767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210574 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9377ef-9af1-4c91-a65e-c831b6874962-config-volume\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.210767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210599 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9377ef-9af1-4c91-a65e-c831b6874962-tmp-dir\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.210767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210637 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdkw\" (UniqueName: \"kubernetes.io/projected/dc9377ef-9af1-4c91-a65e-c831b6874962-kube-api-access-kjdkw\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.210767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.210670 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.210971 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:29.210776 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:29.210971 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:29.210844 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls podName:dc9377ef-9af1-4c91-a65e-c831b6874962 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:29.710825375 +0000 UTC m=+36.529596636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls") pod "dns-default-8w8gb" (UID: "dc9377ef-9af1-4c91-a65e-c831b6874962") : secret "dns-default-metrics-tls" not found Jul 17 18:10:29.211119 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.211000 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc9377ef-9af1-4c91-a65e-c831b6874962-tmp-dir\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.211208 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.211183 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9377ef-9af1-4c91-a65e-c831b6874962-config-volume\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.216311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.216285 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bff42af5-1a78-43de-a2c6-2dabf7412bd7-cert\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.219157 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.219129 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdkw\" (UniqueName: \"kubernetes.io/projected/dc9377ef-9af1-4c91-a65e-c831b6874962-kube-api-access-kjdkw\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.219356 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.219330 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5wk\" (UniqueName: \"kubernetes.io/projected/bff42af5-1a78-43de-a2c6-2dabf7412bd7-kube-api-access-ft5wk\") pod \"ingress-canary-z46x2\" (UID: \"bff42af5-1a78-43de-a2c6-2dabf7412bd7\") " pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.333460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.333420 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z46x2" Jul 17 18:10:29.716366 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.716329 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:29.716529 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:29.716513 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:29.716597 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:29.716586 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls podName:dc9377ef-9af1-4c91-a65e-c831b6874962 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:30.716568364 +0000 UTC m=+37.535339621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls") pod "dns-default-8w8gb" (UID: "dc9377ef-9af1-4c91-a65e-c831b6874962") : secret "dns-default-metrics-tls" not found Jul 17 18:10:29.734379 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.734348 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:29.736377 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.736352 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:10:29.736494 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:29.736467 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nj845\"" Jul 17 18:10:30.284351 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.284176 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z46x2"] Jul 17 18:10:30.349494 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:30.349443 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff42af5_1a78_43de_a2c6_2dabf7412bd7.slice/crio-b88cd0edf253214bf366d95343dd4cd82f82c90698032a0332a9a90719caeb06 WatchSource:0}: Error finding container b88cd0edf253214bf366d95343dd4cd82f82c90698032a0332a9a90719caeb06: Status 404 returned error can't find the container with id b88cd0edf253214bf366d95343dd4cd82f82c90698032a0332a9a90719caeb06 Jul 17 18:10:30.725844 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.725750 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:30.726001 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:30.725909 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:30.726072 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:30.726002 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls podName:dc9377ef-9af1-4c91-a65e-c831b6874962 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:32.725964382 +0000 UTC m=+39.544735627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls") pod "dns-default-8w8gb" (UID: "dc9377ef-9af1-4c91-a65e-c831b6874962") : secret "dns-default-metrics-tls" not found Jul 17 18:10:30.734037 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.734018 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:30.734127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.734039 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:30.736421 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.736400 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:10:30.736633 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.736612 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:10:30.736733 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.736717 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:10:30.736802 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:30.736791 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47jdr\"" Jul 17 18:10:31.062539 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.062509 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="e4bd94fc1cff96d8d951ec5735ab5cfc7986210f700161ac8142705ef35d1e37" exitCode=0 Jul 17 18:10:31.062714 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.062591 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"e4bd94fc1cff96d8d951ec5735ab5cfc7986210f700161ac8142705ef35d1e37"} Jul 17 18:10:31.063655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.063637 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z46x2" event={"ID":"bff42af5-1a78-43de-a2c6-2dabf7412bd7","Type":"ContainerStarted","Data":"b88cd0edf253214bf366d95343dd4cd82f82c90698032a0332a9a90719caeb06"} Jul 17 18:10:31.967525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.967491 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms"] Jul 17 18:10:31.971891 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.971863 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" Jul 17 18:10:31.974188 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.974163 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 17 18:10:31.974711 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.974687 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2fd6j\"" Jul 17 18:10:31.974813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.974715 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 17 18:10:31.978804 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:31.978766 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms"] Jul 17 18:10:32.072701 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.072665 2587 generic.go:356] "Generic (PLEG): container finished" podID="da86941c-593e-4b66-b353-52e1d08c9b99" containerID="d7713915e8738c4a67f3241de707a294b9cda797fe10c08b0e767b4bf695b23c" exitCode=0 Jul 17 18:10:32.072907 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.072747 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerDied","Data":"d7713915e8738c4a67f3241de707a294b9cda797fe10c08b0e767b4bf695b23c"} Jul 17 18:10:32.137208 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.137165 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7qn\" (UniqueName: \"kubernetes.io/projected/7f8943d1-361a-4dee-b7f4-5e97f511fb6a-kube-api-access-ff7qn\") pod \"migrator-7cc9c7ffd7-52rms\" (UID: \"7f8943d1-361a-4dee-b7f4-5e97f511fb6a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" Jul 17 18:10:32.238391 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.238304 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7qn\" (UniqueName: \"kubernetes.io/projected/7f8943d1-361a-4dee-b7f4-5e97f511fb6a-kube-api-access-ff7qn\") pod \"migrator-7cc9c7ffd7-52rms\" (UID: \"7f8943d1-361a-4dee-b7f4-5e97f511fb6a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" Jul 17 18:10:32.248929 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.248895 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7qn\" (UniqueName: \"kubernetes.io/projected/7f8943d1-361a-4dee-b7f4-5e97f511fb6a-kube-api-access-ff7qn\") pod \"migrator-7cc9c7ffd7-52rms\" (UID: \"7f8943d1-361a-4dee-b7f4-5e97f511fb6a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" Jul 17 18:10:32.283738 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.283700 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" Jul 17 18:10:32.666918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.666744 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms"] Jul 17 18:10:32.669531 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:32.669493 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8943d1_361a_4dee_b7f4_5e97f511fb6a.slice/crio-d7fcf04d7741284ad7bf0325e63a805947d0f3297c7bd0ab92de2db69322d2c3 WatchSource:0}: Error finding container d7fcf04d7741284ad7bf0325e63a805947d0f3297c7bd0ab92de2db69322d2c3: Status 404 returned error can't find the container with id d7fcf04d7741284ad7bf0325e63a805947d0f3297c7bd0ab92de2db69322d2c3 Jul 17 18:10:32.689504 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.689474 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n57qg"] Jul 17 18:10:32.692499 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.692482 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.696817 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.696786 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 18:10:32.697314 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.697299 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 18:10:32.697631 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.697607 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85j6j\"" Jul 17 18:10:32.697631 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.697621 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 18:10:32.697869 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.697850 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 18:10:32.707111 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.707028 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n57qg"] Jul 17 18:10:32.743771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.743732 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:32.743969 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.743870 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:32.743969 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.743932 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls podName:dc9377ef-9af1-4c91-a65e-c831b6874962 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:36.743914532 +0000 UTC m=+43.562685781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls") pod "dns-default-8w8gb" (UID: "dc9377ef-9af1-4c91-a65e-c831b6874962") : secret "dns-default-metrics-tls" not found Jul 17 18:10:32.844316 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.844273 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.844489 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.844397 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87654b77-0f45-4df5-ba9e-d6002088b336-data-volume\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.844489 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.844457 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8ts\" (UniqueName: \"kubernetes.io/projected/87654b77-0f45-4df5-ba9e-d6002088b336-kube-api-access-fz8ts\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.844489 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.844477 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87654b77-0f45-4df5-ba9e-d6002088b336-crio-socket\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.844601 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.844521 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.944859 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.944812 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.944859 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.944859 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.945109 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.944941 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:32.945109 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.944973 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:32.945109 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.945036 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:33.445017082 +0000 UTC m=+40.263788328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : configmap "kube-rbac-proxy" not found Jul 17 18:10:32.945109 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:32.945055 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:33.445047447 +0000 UTC m=+40.263818689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:32.945109 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.945070 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87654b77-0f45-4df5-ba9e-d6002088b336-data-volume\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.945320 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.945117 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8ts\" (UniqueName: \"kubernetes.io/projected/87654b77-0f45-4df5-ba9e-d6002088b336-kube-api-access-fz8ts\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.945320 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.945135 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87654b77-0f45-4df5-ba9e-d6002088b336-crio-socket\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.945420 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.945394 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87654b77-0f45-4df5-ba9e-d6002088b336-crio-socket\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.945420 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.945414 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87654b77-0f45-4df5-ba9e-d6002088b336-data-volume\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:32.953761 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:32.953735 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8ts\" (UniqueName: \"kubernetes.io/projected/87654b77-0f45-4df5-ba9e-d6002088b336-kube-api-access-fz8ts\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:33.076305 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.076269 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z46x2" event={"ID":"bff42af5-1a78-43de-a2c6-2dabf7412bd7","Type":"ContainerStarted","Data":"0f891a181b24cba99aa303bcabedeb3b0906d5650a71af5dde9109b32bba3bca"} Jul 17 18:10:33.077724 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.077699 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" event={"ID":"7f8943d1-361a-4dee-b7f4-5e97f511fb6a","Type":"ContainerStarted","Data":"d7fcf04d7741284ad7bf0325e63a805947d0f3297c7bd0ab92de2db69322d2c3"} Jul 17 18:10:33.080862 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.080834 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" event={"ID":"da86941c-593e-4b66-b353-52e1d08c9b99","Type":"ContainerStarted","Data":"664ad9207a8e5f8ba1885681afcf226c81ddb8e5d270a52f833b9191afeae68d"} Jul 17 18:10:33.089892 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.089831 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z46x2" podStartSLOduration=2.89337386 podStartE2EDuration="5.089813041s" totalImagesPullingTime="2.196439181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:33.08924353 +0000 UTC m=+39.908014791" watchObservedRunningTime="2026-07-17 18:10:33.089813041 +0000 UTC m=+39.908584306" Jul 17 18:10:33.111271 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.111172 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xdn5t" podStartSLOduration=7.670223227 podStartE2EDuration="39.111154878s" totalImagesPullingTime="31.440931651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:33.110771952 +0000 UTC m=+39.929543216" watchObservedRunningTime="2026-07-17 18:10:33.111154878 +0000 UTC m=+39.929926141" Jul 17 18:10:33.449670 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.449584 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:33.449670 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:33.449636 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:33.449891 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:33.449708 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:33.449891 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:33.449781 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:33.449891 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:33.449789 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:34.449768232 +0000 UTC m=+41.268539474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : configmap "kube-rbac-proxy" not found Jul 17 18:10:33.449891 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:33.449833 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:34.449817626 +0000 UTC m=+41.268588888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:34.085227 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:34.085187 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" event={"ID":"7f8943d1-361a-4dee-b7f4-5e97f511fb6a","Type":"ContainerStarted","Data":"ecebead93b7e5e8047c61d5a521db0c42625ec35cef9e7aae0501122b661f964"} Jul 17 18:10:34.085587 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:34.085244 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" event={"ID":"7f8943d1-361a-4dee-b7f4-5e97f511fb6a","Type":"ContainerStarted","Data":"f2080fcdeb5ab95d45ba436af1774cf1f33cf6b30fcff0300f8e6df8bd4c4126"} Jul 17 18:10:34.098940 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:34.098842 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-52rms" podStartSLOduration=1.919873825 podStartE2EDuration="3.098828368s" totalImagesPullingTime="1.178954543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:34.098665206 +0000 UTC m=+40.917436472" watchObservedRunningTime="2026-07-17 18:10:34.098828368 +0000 UTC m=+40.917599632" Jul 17 18:10:34.458889 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:34.458808 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:34.459048 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:34.458903 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:34.459048 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:34.458972 2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:10:34.459048 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:34.458975 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:34.459048 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:34.459049 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:36.459034495 +0000 UTC m=+43.277805737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : configmap "kube-rbac-proxy" not found Jul 17 18:10:34.459174 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:34.459089 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:36.459074224 +0000 UTC m=+43.277845475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:35.581244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:35.581216 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-csmjl_e43c7842-165d-4505-b7a3-803c73c7c01d/dns-node-resolver/0.log" Jul 17 18:10:36.474461 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:36.474413 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:36.474461 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:36.474459 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:36.474700 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:36.474581 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:36.474700 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:36.474634 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:40.474621056 +0000 UTC m=+47.293392298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:36.474999 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:36.474959 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87654b77-0f45-4df5-ba9e-d6002088b336-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:36.582161 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:36.582133 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lqzqv_28775f48-6fd5-429e-bf24-5567776b8861/node-ca/0.log" Jul 17 18:10:36.776873 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:36.776839 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:36.777047 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:36.776958 2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:10:36.777047 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:36.777029 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls podName:dc9377ef-9af1-4c91-a65e-c831b6874962 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:44.777015914 +0000 UTC m=+51.595787156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls") pod "dns-default-8w8gb" (UID: "dc9377ef-9af1-4c91-a65e-c831b6874962") : secret "dns-default-metrics-tls" not found Jul 17 18:10:37.382664 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:37.382638 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z46x2_bff42af5-1a78-43de-a2c6-2dabf7412bd7/serve-healthcheck-canary/0.log" Jul 17 18:10:37.583102 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:37.583073 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-52rms_7f8943d1-361a-4dee-b7f4-5e97f511fb6a/migrator/0.log" Jul 17 18:10:37.781809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:37.781781 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-52rms_7f8943d1-361a-4dee-b7f4-5e97f511fb6a/graceful-termination/0.log" Jul 17 18:10:40.505510 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:40.505474 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:40.506024 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:40.505634 2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:10:40.506024 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:10:40.505702 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls podName:87654b77-0f45-4df5-ba9e-d6002088b336 nodeName:}" failed. No retries permitted until 2026-07-17 18:10:48.50568545 +0000 UTC m=+55.324456694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n57qg" (UID: "87654b77-0f45-4df5-ba9e-d6002088b336") : secret "insights-runtime-extractor-tls" not found Jul 17 18:10:44.840277 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:44.840233 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:44.842510 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:44.842475 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc9377ef-9af1-4c91-a65e-c831b6874962-metrics-tls\") pod \"dns-default-8w8gb\" (UID: \"dc9377ef-9af1-4c91-a65e-c831b6874962\") " pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:44.921691 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:44.921653 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:45.050906 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:45.050876 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8w8gb"] Jul 17 18:10:45.054246 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:45.054215 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9377ef_9af1_4c91_a65e_c831b6874962.slice/crio-542b3d3ffc5586a38b176628beedc2b8a6c7e587caad2745c238c427ebd6a16e WatchSource:0}: Error finding container 542b3d3ffc5586a38b176628beedc2b8a6c7e587caad2745c238c427ebd6a16e: Status 404 returned error can't find the container with id 542b3d3ffc5586a38b176628beedc2b8a6c7e587caad2745c238c427ebd6a16e Jul 17 18:10:45.111366 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:45.111269 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8w8gb" event={"ID":"dc9377ef-9af1-4c91-a65e-c831b6874962","Type":"ContainerStarted","Data":"542b3d3ffc5586a38b176628beedc2b8a6c7e587caad2745c238c427ebd6a16e"} Jul 17 18:10:48.122040 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.122003 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8w8gb" event={"ID":"dc9377ef-9af1-4c91-a65e-c831b6874962","Type":"ContainerStarted","Data":"46212476b792beb1b90c7675aba58df40b2e6060edf6ba908ebd9b0e0168cd01"} Jul 17 18:10:48.122499 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.122053 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8w8gb" event={"ID":"dc9377ef-9af1-4c91-a65e-c831b6874962","Type":"ContainerStarted","Data":"063553c52122067215d94d2ba4a4b98348b912b8374bca288921b047610deec3"} Jul 17 18:10:48.122499 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.122084 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:48.139505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.139458 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8w8gb" podStartSLOduration=17.690254108 podStartE2EDuration="20.139441276s" totalImagesPullingTime="2.449187168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:48.138108313 +0000 UTC m=+54.956879578" watchObservedRunningTime="2026-07-17 18:10:48.139441276 +0000 UTC m=+54.958212540" Jul 17 18:10:48.576284 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.576237 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:48.578524 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.578498 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87654b77-0f45-4df5-ba9e-d6002088b336-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n57qg\" (UID: \"87654b77-0f45-4df5-ba9e-d6002088b336\") " pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:48.602513 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.602487 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n57qg" Jul 17 18:10:48.745452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:48.745411 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n57qg"] Jul 17 18:10:48.748305 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:48.748266 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87654b77_0f45_4df5_ba9e_d6002088b336.slice/crio-3898dd9eb09006e234f1fa82a81a697504c95b9a346f576ced5692f2d42443db WatchSource:0}: Error finding container 3898dd9eb09006e234f1fa82a81a697504c95b9a346f576ced5692f2d42443db: Status 404 returned error can't find the container with id 3898dd9eb09006e234f1fa82a81a697504c95b9a346f576ced5692f2d42443db Jul 17 18:10:49.127728 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:49.127686 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n57qg" event={"ID":"87654b77-0f45-4df5-ba9e-d6002088b336","Type":"ContainerStarted","Data":"3898dd9eb09006e234f1fa82a81a697504c95b9a346f576ced5692f2d42443db"} Jul 17 18:10:54.049494 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:54.049471 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljz5r" Jul 17 18:10:54.146544 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:54.146509 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n57qg" event={"ID":"87654b77-0f45-4df5-ba9e-d6002088b336","Type":"ContainerStarted","Data":"b0c0a942c20c2f4b65e9abc8316cd3f3dc4a02f60cffc10c8d505f5d9c26b4b4"} Jul 17 18:10:57.073225 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.073187 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br"] Jul 17 18:10:57.084656 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.084628 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.087647 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.087618 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 18:10:57.087774 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.087658 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 18:10:57.088181 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.088127 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 18:10:57.088181 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.088136 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5h2w7\"" Jul 17 18:10:57.088473 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.088455 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 17 18:10:57.091969 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.091928 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br"] Jul 17 18:10:57.139113 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.139079 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xjc\" (UniqueName: \"kubernetes.io/projected/de7c6d00-073b-4cf6-949d-93b8067c5fc2-kube-api-access-r5xjc\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.139285 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.139134 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de7c6d00-073b-4cf6-949d-93b8067c5fc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.179658 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.179618 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-jgppc"] Jul 17 18:10:57.184108 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.184085 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:10:57.184582 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.184554 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f5587f5bc-ksm26"] Jul 17 18:10:57.186423 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.186399 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 18:10:57.186548 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.186450 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 18:10:57.186716 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.186695 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8dcxr\"" Jul 17 18:10:57.187776 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.187753 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.190282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.190257 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 18:10:57.190379 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.190266 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 18:10:57.191337 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.191265 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksssl\"" Jul 17 18:10:57.198676 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.198656 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 18:10:57.203648 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.203611 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jgppc"] Jul 17 18:10:57.207809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.207771 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5587f5bc-ksm26"] Jul 17 18:10:57.211871 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.211853 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 18:10:57.240363 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240338 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75tb\" (UniqueName: \"kubernetes.io/projected/640f9ff7-4abf-4882-b8e6-1e9225680878-kube-api-access-g75tb\") pod \"downloads-5bc67dcd46-jgppc\" (UID: \"640f9ff7-4abf-4882-b8e6-1e9225680878\") " pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:10:57.240503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240396 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xjc\" (UniqueName: \"kubernetes.io/projected/de7c6d00-073b-4cf6-949d-93b8067c5fc2-kube-api-access-r5xjc\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.240503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240425 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-bound-sa-token\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240466 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-tls\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240626 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240509 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de7c6d00-073b-4cf6-949d-93b8067c5fc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.240626 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240529 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91c3fa1-725a-4c89-8dfd-213791366ae5-ca-trust-extracted\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240626 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240550 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-installation-pull-secrets\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240626 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240577 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-trusted-ca\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240644 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sml5f\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-kube-api-access-sml5f\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240700 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-certificates\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.240771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.240738 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-image-registry-private-configuration\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.243348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.243326 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de7c6d00-073b-4cf6-949d-93b8067c5fc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.263144 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.263115 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xjc\" (UniqueName: \"kubernetes.io/projected/de7c6d00-073b-4cf6-949d-93b8067c5fc2-kube-api-access-r5xjc\") pod \"managed-serviceaccount-addon-agent-55cb8b96d8-pl5br\" (UID: \"de7c6d00-073b-4cf6-949d-93b8067c5fc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.341645 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341576 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-certificates\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341645 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341628 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-image-registry-private-configuration\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341657 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75tb\" (UniqueName: \"kubernetes.io/projected/640f9ff7-4abf-4882-b8e6-1e9225680878-kube-api-access-g75tb\") pod \"downloads-5bc67dcd46-jgppc\" (UID: \"640f9ff7-4abf-4882-b8e6-1e9225680878\") " pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341689 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-bound-sa-token\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341713 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-tls\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341743 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91c3fa1-725a-4c89-8dfd-213791366ae5-ca-trust-extracted\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341768 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-installation-pull-secrets\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341794 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-trusted-ca\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.341952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.341837 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sml5f\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-kube-api-access-sml5f\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.342315 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.342203 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91c3fa1-725a-4c89-8dfd-213791366ae5-ca-trust-extracted\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.342693 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.342661 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-certificates\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.343534 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.343508 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91c3fa1-725a-4c89-8dfd-213791366ae5-trusted-ca\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.344572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.344552 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-image-registry-private-configuration\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.344655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.344581 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-registry-tls\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.344655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.344643 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91c3fa1-725a-4c89-8dfd-213791366ae5-installation-pull-secrets\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.352074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.352039 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75tb\" (UniqueName: \"kubernetes.io/projected/640f9ff7-4abf-4882-b8e6-1e9225680878-kube-api-access-g75tb\") pod \"downloads-5bc67dcd46-jgppc\" (UID: \"640f9ff7-4abf-4882-b8e6-1e9225680878\") " pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:10:57.352572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.352549 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sml5f\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-kube-api-access-sml5f\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.353589 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.353573 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91c3fa1-725a-4c89-8dfd-213791366ae5-bound-sa-token\") pod \"image-registry-f5587f5bc-ksm26\" (UID: \"a91c3fa1-725a-4c89-8dfd-213791366ae5\") " pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.407412 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.407378 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" Jul 17 18:10:57.495592 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.495549 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:10:57.501483 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.501314 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:57.567466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.566800 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br"] Jul 17 18:10:57.633877 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.633849 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jgppc"] Jul 17 18:10:57.637029 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:57.637001 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640f9ff7_4abf_4882_b8e6_1e9225680878.slice/crio-0f1d161d96885222f4c10a6972cded73b827f2ea46ca8b8b84e43505e3d1c2d4 WatchSource:0}: Error finding container 0f1d161d96885222f4c10a6972cded73b827f2ea46ca8b8b84e43505e3d1c2d4: Status 404 returned error can't find the container with id 0f1d161d96885222f4c10a6972cded73b827f2ea46ca8b8b84e43505e3d1c2d4 Jul 17 18:10:57.651027 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:57.650977 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5587f5bc-ksm26"] Jul 17 18:10:57.653633 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:57.653609 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91c3fa1_725a_4c89_8dfd_213791366ae5.slice/crio-15f0f6577efa1ed4db61e3fa4247aa672a24544bae902814476d47ee384bc61b WatchSource:0}: Error finding container 15f0f6577efa1ed4db61e3fa4247aa672a24544bae902814476d47ee384bc61b: Status 404 returned error can't find the container with id 15f0f6577efa1ed4db61e3fa4247aa672a24544bae902814476d47ee384bc61b Jul 17 18:10:58.130167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.130137 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8w8gb" Jul 17 18:10:58.163162 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.163125 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" event={"ID":"a91c3fa1-725a-4c89-8dfd-213791366ae5","Type":"ContainerStarted","Data":"cbbfadadd8561813aec5025fef3cdd41fd8d5b8856bcc115099c4d31c1b1f61a"} Jul 17 18:10:58.163335 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.163183 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" event={"ID":"a91c3fa1-725a-4c89-8dfd-213791366ae5","Type":"ContainerStarted","Data":"15f0f6577efa1ed4db61e3fa4247aa672a24544bae902814476d47ee384bc61b"} Jul 17 18:10:58.163335 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.163274 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:10:58.164609 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.164583 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jgppc" event={"ID":"640f9ff7-4abf-4882-b8e6-1e9225680878","Type":"ContainerStarted","Data":"0f1d161d96885222f4c10a6972cded73b827f2ea46ca8b8b84e43505e3d1c2d4"} Jul 17 18:10:58.166133 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.166094 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" event={"ID":"de7c6d00-073b-4cf6-949d-93b8067c5fc2","Type":"ContainerStarted","Data":"da67b03b27aae31e5f78fefa33bb4b1dfe7116126daf4814e4a97b8a0be3fb08"} Jul 17 18:10:58.168139 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.168112 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n57qg" event={"ID":"87654b77-0f45-4df5-ba9e-d6002088b336","Type":"ContainerStarted","Data":"4f5c8fdf1447440e41d90403ae0a3806560b6b1b97874d5902669bef0d0801c9"} Jul 17 18:10:58.181164 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.181113 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" podStartSLOduration=1.18109409 podStartE2EDuration="1.18109409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:10:58.179723307 +0000 UTC m=+64.998494572" watchObservedRunningTime="2026-07-17 18:10:58.18109409 +0000 UTC m=+64.999865333" Jul 17 18:10:58.451752 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.451153 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:58.451752 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.451248 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:58.453728 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.453166 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:10:58.453728 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.453441 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:10:58.468331 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.468296 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73cf7878-9139-48a9-a8ea-5c958c28dd3d-original-pull-secret\") pod \"global-pull-secret-syncer-4nfq7\" (UID: \"73cf7878-9139-48a9-a8ea-5c958c28dd3d\") " pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:58.470741 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.470704 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a40295-d82e-413d-b438-e0b99b4f5fdc-metrics-certs\") pod \"network-metrics-daemon-q6mz4\" (UID: \"02a40295-d82e-413d-b438-e0b99b4f5fdc\") " pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:58.547124 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.547094 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nj845\"" Jul 17 18:10:58.551939 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.551911 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:58.553697 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.553669 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:10:58.555199 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.555177 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6mz4" Jul 17 18:10:58.564192 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.564172 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:10:58.577139 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.577117 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h28c\" (UniqueName: \"kubernetes.io/projected/9150e77e-d628-426c-8b6c-791aee78fd73-kube-api-access-6h28c\") pod \"network-check-target-9lcq9\" (UID: \"9150e77e-d628-426c-8b6c-791aee78fd73\") " pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:58.646230 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.646010 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47jdr\"" Jul 17 18:10:58.648853 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.648673 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4nfq7" Jul 17 18:10:58.654571 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.654518 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:10:58.739885 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.739735 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q6mz4"] Jul 17 18:10:58.754074 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:58.754019 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a40295_d82e_413d_b438_e0b99b4f5fdc.slice/crio-b4dec0b89212b07b2adcd5ad48e49c392d0e93fc51c893fd92816bd723efb405 WatchSource:0}: Error finding container b4dec0b89212b07b2adcd5ad48e49c392d0e93fc51c893fd92816bd723efb405: Status 404 returned error can't find the container with id b4dec0b89212b07b2adcd5ad48e49c392d0e93fc51c893fd92816bd723efb405 Jul 17 18:10:58.855210 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.855125 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4nfq7"] Jul 17 18:10:58.861855 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:58.861813 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cf7878_9139_48a9_a8ea_5c958c28dd3d.slice/crio-6eeb959f3ac5d09af0527be80348bc2173beed7ce6b4e53039ae29329d30749d WatchSource:0}: Error finding container 6eeb959f3ac5d09af0527be80348bc2173beed7ce6b4e53039ae29329d30749d: Status 404 returned error can't find the container with id 6eeb959f3ac5d09af0527be80348bc2173beed7ce6b4e53039ae29329d30749d Jul 17 18:10:58.876559 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:58.876491 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9lcq9"] Jul 17 18:10:58.882760 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:10:58.882718 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9150e77e_d628_426c_8b6c_791aee78fd73.slice/crio-bb26321f4369c6f0044e54051e499bd7f880934bbce4db93990d78f57bcec085 WatchSource:0}: Error finding container bb26321f4369c6f0044e54051e499bd7f880934bbce4db93990d78f57bcec085: Status 404 returned error can't find the container with id bb26321f4369c6f0044e54051e499bd7f880934bbce4db93990d78f57bcec085 Jul 17 18:10:59.173611 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:59.173563 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9lcq9" event={"ID":"9150e77e-d628-426c-8b6c-791aee78fd73","Type":"ContainerStarted","Data":"bb26321f4369c6f0044e54051e499bd7f880934bbce4db93990d78f57bcec085"} Jul 17 18:10:59.175095 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:59.175023 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4nfq7" event={"ID":"73cf7878-9139-48a9-a8ea-5c958c28dd3d","Type":"ContainerStarted","Data":"6eeb959f3ac5d09af0527be80348bc2173beed7ce6b4e53039ae29329d30749d"} Jul 17 18:10:59.177284 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:10:59.177259 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6mz4" event={"ID":"02a40295-d82e-413d-b438-e0b99b4f5fdc","Type":"ContainerStarted","Data":"b4dec0b89212b07b2adcd5ad48e49c392d0e93fc51c893fd92816bd723efb405"} Jul 17 18:11:01.641016 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.640915 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:01.645393 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.645128 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.647355 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647142 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 18:11:01.647838 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647747 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 18:11:01.647838 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647778 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 18:11:01.647838 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647813 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrbhg\"" Jul 17 18:11:01.648095 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647813 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 18:11:01.648095 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.647899 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 18:11:01.658324 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.658294 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678476 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678523 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vhj\" (UniqueName: \"kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678553 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678582 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678613 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.678795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.678677 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.765694 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.764584 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l"] Jul 17 18:11:01.769344 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.769310 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:01.771780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.771565 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 18:11:01.771780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.771604 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tnk9w\"" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.778394 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l"] Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779035 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779092 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779118 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vhj\" (UniqueName: \"kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779141 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779167 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779191 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.780010 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.779939 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.783622 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.783566 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.783764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.783737 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.786648 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.786597 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.788105 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.788049 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.801748 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.801714 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vhj\" (UniqueName: \"kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj\") pod \"console-698bff8fb7-chh4c\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.879808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.879757 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-b2v6l\" (UID: \"aa6cbf55-c59e-4862-b10e-53f7d881f3ba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:01.961944 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.961785 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:01.980661 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:01.980621 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-b2v6l\" (UID: \"aa6cbf55-c59e-4862-b10e-53f7d881f3ba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:01.980821 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:11:01.980780 2587 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 17 18:11:01.980885 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:11:01.980851 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates podName:aa6cbf55-c59e-4862-b10e-53f7d881f3ba nodeName:}" failed. No retries permitted until 2026-07-17 18:11:02.480829767 +0000 UTC m=+69.299601011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-b2v6l" (UID: "aa6cbf55-c59e-4862-b10e-53f7d881f3ba") : secret "prometheus-operator-admission-webhook-tls" not found Jul 17 18:11:02.485696 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:02.485601 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-b2v6l\" (UID: \"aa6cbf55-c59e-4862-b10e-53f7d881f3ba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:02.488732 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:02.488701 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aa6cbf55-c59e-4862-b10e-53f7d881f3ba-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-b2v6l\" (UID: \"aa6cbf55-c59e-4862-b10e-53f7d881f3ba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:02.704220 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:02.704180 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:07.143184 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.143118 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:07.177510 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.177372 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l"] Jul 17 18:11:07.218547 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.217734 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9lcq9" event={"ID":"9150e77e-d628-426c-8b6c-791aee78fd73","Type":"ContainerStarted","Data":"e78acba799d8e951cd789878f85081e38124cb25cbb053d2c329458438e5401c"} Jul 17 18:11:07.218547 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.218498 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:11:07.221373 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.220497 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" event={"ID":"aa6cbf55-c59e-4862-b10e-53f7d881f3ba","Type":"ContainerStarted","Data":"4129d5aef640e213d8f417c92407c6775ae2368aeb4750e2edd45c73136e3c5e"} Jul 17 18:11:07.223613 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.223370 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" event={"ID":"de7c6d00-073b-4cf6-949d-93b8067c5fc2","Type":"ContainerStarted","Data":"4cc5888f84a583ac0899241743b73efc1b2e6e0728bb55d7ed40ab2f9a7d18b6"} Jul 17 18:11:07.226024 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.225954 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6mz4" event={"ID":"02a40295-d82e-413d-b438-e0b99b4f5fdc","Type":"ContainerStarted","Data":"a660eb0bd8383675fde914b33a45c52f97e14968f9aa63ea259acda4cf3c1e06"} Jul 17 18:11:07.227371 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.227343 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698bff8fb7-chh4c" event={"ID":"5379773a-ba43-468c-bc21-24555bafaa31","Type":"ContainerStarted","Data":"5f50831c2abf36f4ac7f96b8aa195a29c4e7bc9acfc3233d7798668fb1e402fe"} Jul 17 18:11:07.233538 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.233031 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n57qg" event={"ID":"87654b77-0f45-4df5-ba9e-d6002088b336","Type":"ContainerStarted","Data":"4c327d413c896acc6493729ef06bd1e05f1166f8844fb9abf013743bcae7a97a"} Jul 17 18:11:07.233829 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.233710 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9lcq9" podStartSLOduration=65.126880946 podStartE2EDuration="1m13.233693291s" totalImagesPullingTime="8.106812345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:07.232472473 +0000 UTC m=+74.051243737" watchObservedRunningTime="2026-07-17 18:11:07.233693291 +0000 UTC m=+74.052464557" Jul 17 18:11:07.247940 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:07.247785 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55cb8b96d8-pl5br" podStartSLOduration=0.836741224 podStartE2EDuration="10.247765522s" totalImagesPullingTime="9.411024298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:07.247194437 +0000 UTC m=+74.065965701" watchObservedRunningTime="2026-07-17 18:11:07.247765522 +0000 UTC m=+74.066536804" Jul 17 18:11:08.240863 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:08.240185 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4nfq7" event={"ID":"73cf7878-9139-48a9-a8ea-5c958c28dd3d","Type":"ContainerStarted","Data":"045d3d19b5080b455eeb0c1244b76f01f9800ba40236a49d60efd86b15f71c59"} Jul 17 18:11:08.244942 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:08.244898 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6mz4" event={"ID":"02a40295-d82e-413d-b438-e0b99b4f5fdc","Type":"ContainerStarted","Data":"06afaefd3e351989548fd31c8216c23138a348a15eeef2351262b84962c2a9bf"} Jul 17 18:11:08.258006 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:08.257665 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n57qg" podStartSLOduration=18.142067872 podStartE2EDuration="36.257644644s" totalImagesPullingTime="18.115576772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:10:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:07.268158797 +0000 UTC m=+74.086930064" watchObservedRunningTime="2026-07-17 18:11:08.257644644 +0000 UTC m=+75.076415910" Jul 17 18:11:08.258006 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:08.257801 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4nfq7" podStartSLOduration=65.964648913 podStartE2EDuration="1m14.257791622s" totalImagesPullingTime="8.293142709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:08.256788383 +0000 UTC m=+75.075559649" watchObservedRunningTime="2026-07-17 18:11:08.257791622 +0000 UTC m=+75.076562888" Jul 17 18:11:08.272911 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:08.272852 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q6mz4" podStartSLOduration=66.053588637 podStartE2EDuration="1m14.272831896s" totalImagesPullingTime="8.219243259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:08.272525903 +0000 UTC m=+75.091297181" watchObservedRunningTime="2026-07-17 18:11:08.272831896 +0000 UTC m=+75.091603160" Jul 17 18:11:09.249141 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:09.249096 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" event={"ID":"aa6cbf55-c59e-4862-b10e-53f7d881f3ba","Type":"ContainerStarted","Data":"8a719824ff8a6c6a0e26dd721575d9253ac88362c21c205b8ea75051e3e29058"} Jul 17 18:11:09.263708 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:09.263649 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" podStartSLOduration=6.934011476 podStartE2EDuration="8.263631618s" totalImagesPullingTime="1.329620142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:09.262729632 +0000 UTC m=+76.081500888" watchObservedRunningTime="2026-07-17 18:11:09.263631618 +0000 UTC m=+76.082402884" Jul 17 18:11:10.253133 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.253100 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:10.258620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.258596 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-b2v6l" Jul 17 18:11:10.830128 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.830095 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9"] Jul 17 18:11:10.833481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.833453 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:10.835269 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.835245 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 18:11:10.835680 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.835652 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 18:11:10.836201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.836180 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 18:11:10.836329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.836308 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 18:11:10.836392 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.836321 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ljwf4\"" Jul 17 18:11:10.836481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.836463 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 18:11:10.842556 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.842537 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9"] Jul 17 18:11:10.951282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.951224 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5998763a-0cce-4fcb-b486-ab98a59b2d74-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:10.951460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.951293 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:10.951460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.951326 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:10.951460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:10.951405 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb99n\" (UniqueName: \"kubernetes.io/projected/5998763a-0cce-4fcb-b486-ab98a59b2d74-kube-api-access-qb99n\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.052495 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.052458 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.052646 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.052503 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb99n\" (UniqueName: \"kubernetes.io/projected/5998763a-0cce-4fcb-b486-ab98a59b2d74-kube-api-access-qb99n\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.052646 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.052554 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5998763a-0cce-4fcb-b486-ab98a59b2d74-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.052646 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.052593 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.053393 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.053359 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5998763a-0cce-4fcb-b486-ab98a59b2d74-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.055202 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.055180 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.055301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.055257 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5998763a-0cce-4fcb-b486-ab98a59b2d74-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.063966 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.063943 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb99n\" (UniqueName: \"kubernetes.io/projected/5998763a-0cce-4fcb-b486-ab98a59b2d74-kube-api-access-qb99n\") pod \"prometheus-operator-5945b5cfd-4jrs9\" (UID: \"5998763a-0cce-4fcb-b486-ab98a59b2d74\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:11.144894 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:11.144799 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" Jul 17 18:11:17.731218 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:17.731121 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9"] Jul 17 18:11:17.733209 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:17.733184 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5998763a_0cce_4fcb_b486_ab98a59b2d74.slice/crio-f9062a73bf8c30090ec9075e18322fad986d7fe9e8f72b19c3111d0db4d53246 WatchSource:0}: Error finding container f9062a73bf8c30090ec9075e18322fad986d7fe9e8f72b19c3111d0db4d53246: Status 404 returned error can't find the container with id f9062a73bf8c30090ec9075e18322fad986d7fe9e8f72b19c3111d0db4d53246 Jul 17 18:11:18.285918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.285875 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" event={"ID":"5998763a-0cce-4fcb-b486-ab98a59b2d74","Type":"ContainerStarted","Data":"f9062a73bf8c30090ec9075e18322fad986d7fe9e8f72b19c3111d0db4d53246"} Jul 17 18:11:18.288127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.288096 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jgppc" event={"ID":"640f9ff7-4abf-4882-b8e6-1e9225680878","Type":"ContainerStarted","Data":"c52f3ee263640cb68af78b6117a3795319bb4863def5a425edcd98c49650b692"} Jul 17 18:11:18.288537 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.288519 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:11:18.291182 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.291130 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698bff8fb7-chh4c" event={"ID":"5379773a-ba43-468c-bc21-24555bafaa31","Type":"ContainerStarted","Data":"42753cd077bdb47f1fd97b7534bce80064e25b8f459532817988fba48349984c"} Jul 17 18:11:18.301117 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.301091 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-jgppc" Jul 17 18:11:18.304647 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.304568 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-jgppc" podStartSLOduration=1.322123652 podStartE2EDuration="21.304552606s" totalImagesPullingTime="19.982428954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:18.304223463 +0000 UTC m=+85.122994727" watchObservedRunningTime="2026-07-17 18:11:18.304552606 +0000 UTC m=+85.123323874" Jul 17 18:11:18.341655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:18.341448 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-698bff8fb7-chh4c" podStartSLOduration=6.889035681 podStartE2EDuration="17.341424894s" totalImagesPullingTime="10.452389213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:18.341118764 +0000 UTC m=+85.159890029" watchObservedRunningTime="2026-07-17 18:11:18.341424894 +0000 UTC m=+85.160196198" Jul 17 18:11:19.182268 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:19.182236 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f5587f5bc-ksm26" Jul 17 18:11:19.710593 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:19.710561 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:20.300747 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:20.300708 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" event={"ID":"5998763a-0cce-4fcb-b486-ab98a59b2d74","Type":"ContainerStarted","Data":"993be770111c48ef520939d77280b49d4e794ebc07ffdd4a3c79ede5db19340d"} Jul 17 18:11:20.301254 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:20.300809 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" event={"ID":"5998763a-0cce-4fcb-b486-ab98a59b2d74","Type":"ContainerStarted","Data":"64e612af4de51e5790933b2e98404d5d22f3e6f553b7522ac2a3be79cb588827"} Jul 17 18:11:20.329937 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:20.329875 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-4jrs9" podStartSLOduration=8.478827348 podStartE2EDuration="10.32985399s" totalImagesPullingTime="1.851026642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:20.329024756 +0000 UTC m=+87.147796021" watchObservedRunningTime="2026-07-17 18:11:20.32985399 +0000 UTC m=+87.148625253" Jul 17 18:11:21.962705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:21.962670 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:22.216122 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.216034 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz"] Jul 17 18:11:22.263927 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.263892 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz"] Jul 17 18:11:22.264112 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.263940 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2h5d2"] Jul 17 18:11:22.264112 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.264087 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.267123 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.267097 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 18:11:22.267729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.267709 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 18:11:22.268733 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.268712 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kzl8k\"" Jul 17 18:11:22.286099 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.286071 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.288842 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.288816 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 18:11:22.288975 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.288816 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 18:11:22.289210 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.289181 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r66sj\"" Jul 17 18:11:22.289404 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.289386 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 18:11:22.344554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344517 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d46514e-aa5b-4682-9b23-503119a62222-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.344554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344560 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-wtmp\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.344809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344592 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.344809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344650 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-textfile\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.344809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344694 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.344809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344738 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-root\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.344809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344760 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344815 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-metrics-client-ca\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344839 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-sys\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344886 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344933 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhv4\" (UniqueName: \"kubernetes.io/projected/2d46514e-aa5b-4682-9b23-503119a62222-kube-api-access-9dhv4\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.344969 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqrt\" (UniqueName: \"kubernetes.io/projected/740e8180-4df5-44d8-b136-1e5265fc0aa7-kube-api-access-xhqrt\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.345070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.345013 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.445580 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445540 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.445780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445621 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhv4\" (UniqueName: \"kubernetes.io/projected/2d46514e-aa5b-4682-9b23-503119a62222-kube-api-access-9dhv4\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.445780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445659 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqrt\" (UniqueName: \"kubernetes.io/projected/740e8180-4df5-44d8-b136-1e5265fc0aa7-kube-api-access-xhqrt\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.445780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445690 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.445780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445718 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d46514e-aa5b-4682-9b23-503119a62222-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.445973 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.445919 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-wtmp\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446056 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446011 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.446056 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446044 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-textfile\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446068 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446109 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-root\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446131 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446181 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-metrics-client-ca\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446216 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-sys\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446226 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-wtmp\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446293 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-sys\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446311 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446305 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/740e8180-4df5-44d8-b136-1e5265fc0aa7-root\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446323 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446325 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-textfile\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.446605 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:11:22.446384 2587 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 18:11:22.446605 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:11:22.446440 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls podName:740e8180-4df5-44d8-b136-1e5265fc0aa7 nodeName:}" failed. No retries permitted until 2026-07-17 18:11:22.946420727 +0000 UTC m=+89.765191976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls") pod "node-exporter-2h5d2" (UID: "740e8180-4df5-44d8-b136-1e5265fc0aa7") : secret "node-exporter-tls" not found Jul 17 18:11:22.446605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446455 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d46514e-aa5b-4682-9b23-503119a62222-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.446856 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.446756 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740e8180-4df5-44d8-b136-1e5265fc0aa7-metrics-client-ca\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.448889 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.448866 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.449126 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.449102 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.449206 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.449188 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d46514e-aa5b-4682-9b23-503119a62222-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.454368 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.454342 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhv4\" (UniqueName: \"kubernetes.io/projected/2d46514e-aa5b-4682-9b23-503119a62222-kube-api-access-9dhv4\") pod \"openshift-state-metrics-79f6958cdf-p95fz\" (UID: \"2d46514e-aa5b-4682-9b23-503119a62222\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.454536 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.454381 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqrt\" (UniqueName: \"kubernetes.io/projected/740e8180-4df5-44d8-b136-1e5265fc0aa7-kube-api-access-xhqrt\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.580131 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.580092 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" Jul 17 18:11:22.763215 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.763176 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz"] Jul 17 18:11:22.766047 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:22.766016 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d46514e_aa5b_4682_9b23_503119a62222.slice/crio-298bd803a1f3df606c5b77b70ad5cbfe65088a6513d5020bc08e8ef6f38eb634 WatchSource:0}: Error finding container 298bd803a1f3df606c5b77b70ad5cbfe65088a6513d5020bc08e8ef6f38eb634: Status 404 returned error can't find the container with id 298bd803a1f3df606c5b77b70ad5cbfe65088a6513d5020bc08e8ef6f38eb634 Jul 17 18:11:22.951458 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.951430 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:22.954159 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:22.954124 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/740e8180-4df5-44d8-b136-1e5265fc0aa7-node-exporter-tls\") pod \"node-exporter-2h5d2\" (UID: \"740e8180-4df5-44d8-b136-1e5265fc0aa7\") " pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:23.197752 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:23.197671 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2h5d2" Jul 17 18:11:23.229263 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:23.229228 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740e8180_4df5_44d8_b136_1e5265fc0aa7.slice/crio-2170dd0c7d11e70c024d1687c475b8f3a8baf55c42cb5b21c370b1bb2c9327fd WatchSource:0}: Error finding container 2170dd0c7d11e70c024d1687c475b8f3a8baf55c42cb5b21c370b1bb2c9327fd: Status 404 returned error can't find the container with id 2170dd0c7d11e70c024d1687c475b8f3a8baf55c42cb5b21c370b1bb2c9327fd Jul 17 18:11:23.315282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:23.315178 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h5d2" event={"ID":"740e8180-4df5-44d8-b136-1e5265fc0aa7","Type":"ContainerStarted","Data":"2170dd0c7d11e70c024d1687c475b8f3a8baf55c42cb5b21c370b1bb2c9327fd"} Jul 17 18:11:23.317883 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:23.317848 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" event={"ID":"2d46514e-aa5b-4682-9b23-503119a62222","Type":"ContainerStarted","Data":"d8379f72bfbf6b4fa225683279b2677a1a57ddbab580b8b19712f0a666f3084a"} Jul 17 18:11:23.318023 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:23.317908 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" event={"ID":"2d46514e-aa5b-4682-9b23-503119a62222","Type":"ContainerStarted","Data":"ddca441aab750a32124ca9fd11b2a15921e1258cbe68dc1b4378668c248da0aa"} Jul 17 18:11:23.318023 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:23.317931 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" event={"ID":"2d46514e-aa5b-4682-9b23-503119a62222","Type":"ContainerStarted","Data":"298bd803a1f3df606c5b77b70ad5cbfe65088a6513d5020bc08e8ef6f38eb634"} Jul 17 18:11:24.191739 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.191701 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-556b84bb4f-6zjqt"] Jul 17 18:11:24.231302 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.231269 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-556b84bb4f-6zjqt"] Jul 17 18:11:24.231806 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.231439 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.234394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.233907 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 18:11:24.234394 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.234231 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 18:11:24.234952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.234608 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-agg4e667frbd1\"" Jul 17 18:11:24.234952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.234708 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 18:11:24.234952 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.234790 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 18:11:24.235167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.234976 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 18:11:24.238021 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.236812 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lr984\"" Jul 17 18:11:24.364469 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364416 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4j7m\" (UniqueName: \"kubernetes.io/projected/90ecdd24-7814-494b-8674-6ef55ae965e8-kube-api-access-t4j7m\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364641 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364478 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-grpc-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364641 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364585 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364641 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364627 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364805 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364667 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ecdd24-7814-494b-8674-6ef55ae965e8-metrics-client-ca\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364805 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364699 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364805 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364726 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.364947 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.364810 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465438 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465492 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465534 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ecdd24-7814-494b-8674-6ef55ae965e8-metrics-client-ca\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465565 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465596 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465638 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465690 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4j7m\" (UniqueName: \"kubernetes.io/projected/90ecdd24-7814-494b-8674-6ef55ae965e8-kube-api-access-t4j7m\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.465797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.465716 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-grpc-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.466461 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.466426 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ecdd24-7814-494b-8674-6ef55ae965e8-metrics-client-ca\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.468832 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.468802 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.469405 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.469354 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.470904 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.470678 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-grpc-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.471533 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.471471 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.472293 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.472269 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-tls\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.473424 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.473374 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90ecdd24-7814-494b-8674-6ef55ae965e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.475018 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.474939 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4j7m\" (UniqueName: \"kubernetes.io/projected/90ecdd24-7814-494b-8674-6ef55ae965e8-kube-api-access-t4j7m\") pod \"thanos-querier-556b84bb4f-6zjqt\" (UID: \"90ecdd24-7814-494b-8674-6ef55ae965e8\") " pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:24.550162 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:24.550051 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:25.317108 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:25.316904 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-556b84bb4f-6zjqt"] Jul 17 18:11:25.320041 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:25.320016 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ecdd24_7814_494b_8674_6ef55ae965e8.slice/crio-8f732f9a5985259dce85fdd0ac2b407b31c20c16b7d917458030f9db5a5d4539 WatchSource:0}: Error finding container 8f732f9a5985259dce85fdd0ac2b407b31c20c16b7d917458030f9db5a5d4539: Status 404 returned error can't find the container with id 8f732f9a5985259dce85fdd0ac2b407b31c20c16b7d917458030f9db5a5d4539 Jul 17 18:11:25.333143 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:25.333101 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h5d2" event={"ID":"740e8180-4df5-44d8-b136-1e5265fc0aa7","Type":"ContainerStarted","Data":"5adccb6cb9e2c5d362741cbe91852043773356b3914953bedc2d82b8a06473fb"} Jul 17 18:11:25.337142 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:25.336525 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" event={"ID":"2d46514e-aa5b-4682-9b23-503119a62222","Type":"ContainerStarted","Data":"fc0a162e94760b4827aa42225ef6a3b0b11b9feec13a89ee85a91e0cedeea216"} Jul 17 18:11:25.338616 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:25.338576 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"8f732f9a5985259dce85fdd0ac2b407b31c20c16b7d917458030f9db5a5d4539"} Jul 17 18:11:25.366679 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:25.366578 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-p95fz" podStartSLOduration=1.2306297609999999 podStartE2EDuration="3.366560448s" totalImagesPullingTime="2.135930687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:25.365642415 +0000 UTC m=+92.184413680" watchObservedRunningTime="2026-07-17 18:11:25.366560448 +0000 UTC m=+92.185331713" Jul 17 18:11:26.346104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.346070 2587 generic.go:356] "Generic (PLEG): container finished" podID="740e8180-4df5-44d8-b136-1e5265fc0aa7" containerID="5adccb6cb9e2c5d362741cbe91852043773356b3914953bedc2d82b8a06473fb" exitCode=0 Jul 17 18:11:26.346104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.346152 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h5d2" event={"ID":"740e8180-4df5-44d8-b136-1e5265fc0aa7","Type":"ContainerDied","Data":"5adccb6cb9e2c5d362741cbe91852043773356b3914953bedc2d82b8a06473fb"} Jul 17 18:11:26.627955 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.627924 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c599bf6db-9h9cw"] Jul 17 18:11:26.643580 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.643544 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c599bf6db-9h9cw"] Jul 17 18:11:26.643720 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.643662 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.646481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646179 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 18:11:26.646481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646218 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 18:11:26.646481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646248 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5sr3lfijacf6l\"" Jul 17 18:11:26.646481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646285 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 18:11:26.646481 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646400 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 18:11:26.646763 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.646620 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jmwq\"" Jul 17 18:11:26.786835 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.786791 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787068 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.786846 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-metrics-server-audit-profiles\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787068 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.786923 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz72f\" (UniqueName: \"kubernetes.io/projected/c171558a-c4c4-417d-bd00-9624f79cb9ae-kube-api-access-cz72f\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787068 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.787030 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-tls\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787198 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.787090 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-client-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787198 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.787127 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c171558a-c4c4-417d-bd00-9624f79cb9ae-audit-log\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.787269 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.787214 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-client-certs\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888210 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888129 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-client-certs\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888210 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888173 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888210 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888193 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-metrics-server-audit-profiles\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888224 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz72f\" (UniqueName: \"kubernetes.io/projected/c171558a-c4c4-417d-bd00-9624f79cb9ae-kube-api-access-cz72f\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888269 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-tls\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888318 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-client-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888351 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c171558a-c4c4-417d-bd00-9624f79cb9ae-audit-log\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.888714 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.888692 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c171558a-c4c4-417d-bd00-9624f79cb9ae-audit-log\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.889071 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.889042 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.889281 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.889263 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c171558a-c4c4-417d-bd00-9624f79cb9ae-metrics-server-audit-profiles\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.890709 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.890687 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-client-certs\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.890972 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.890951 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-client-ca-bundle\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.891038 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.890951 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c171558a-c4c4-417d-bd00-9624f79cb9ae-secret-metrics-server-tls\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.895948 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.895929 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz72f\" (UniqueName: \"kubernetes.io/projected/c171558a-c4c4-417d-bd00-9624f79cb9ae-kube-api-access-cz72f\") pod \"metrics-server-5c599bf6db-9h9cw\" (UID: \"c171558a-c4c4-417d-bd00-9624f79cb9ae\") " pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.956257 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.956220 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:26.958151 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.958125 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw"] Jul 17 18:11:26.976384 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.976357 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw"] Jul 17 18:11:26.976502 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.976457 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:26.978454 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.978435 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-287t4\"" Jul 17 18:11:26.978720 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:26.978700 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 18:11:27.089836 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.089799 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrtz\" (UniqueName: \"kubernetes.io/projected/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-kube-api-access-hbrtz\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.090021 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.089850 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.190570 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.190472 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrtz\" (UniqueName: \"kubernetes.io/projected/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-kube-api-access-hbrtz\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.190570 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.190539 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.194032 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.193965 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.198516 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.198487 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrtz\" (UniqueName: \"kubernetes.io/projected/b8af67c7-2a92-4f12-80a9-4663c3d6a19b-kube-api-access-hbrtz\") pod \"monitoring-plugin-7b7b7bc8bf-t4fdw\" (UID: \"b8af67c7-2a92-4f12-80a9-4663c3d6a19b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.288401 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.288363 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:27.353272 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.353236 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h5d2" event={"ID":"740e8180-4df5-44d8-b136-1e5265fc0aa7","Type":"ContainerStarted","Data":"631e4ef20e145d7eb6508e5dc253bd79a8ba3520b7e0c7e92368665a703fe200"} Jul 17 18:11:27.353667 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.353302 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h5d2" event={"ID":"740e8180-4df5-44d8-b136-1e5265fc0aa7","Type":"ContainerStarted","Data":"aab385445a00e940421e3b2b25894b6120b53fe8f409bd37171b2fde064c2728"} Jul 17 18:11:27.375318 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.375268 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2h5d2" podStartSLOduration=3.335305517 podStartE2EDuration="5.375247976s" totalImagesPullingTime="1.925858908s" totalInitContainerRuntime="114.083551ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:27.374366966 +0000 UTC m=+94.193138230" watchObservedRunningTime="2026-07-17 18:11:27.375247976 +0000 UTC m=+94.194019242" Jul 17 18:11:27.457615 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.457529 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-755bfb49b6-lzz5v"] Jul 17 18:11:27.478458 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.478425 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-755bfb49b6-lzz5v"] Jul 17 18:11:27.478615 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.478515 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.480861 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.480835 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 18:11:27.481009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.480869 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 18:11:27.481009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.480914 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 18:11:27.481009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.481003 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 18:11:27.481185 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.481027 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s57nc\"" Jul 17 18:11:27.481236 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.481220 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 18:11:27.486860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.486674 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594536 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bqh\" (UniqueName: \"kubernetes.io/projected/0f535c45-1476-4127-bd45-6b33e94405f1-kube-api-access-74bqh\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594615 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594652 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594689 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594725 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-serving-certs-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.594770 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594763 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-metrics-client-ca\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.595237 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594831 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.595237 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.594868 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-federate-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.674395 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.674372 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw"] Jul 17 18:11:27.677369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:27.677343 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8af67c7_2a92_4f12_80a9_4663c3d6a19b.slice/crio-bdcf4430ffda30b0ffa24fbb91f03c35c4b78da72e73571b9d4cb1640ef582a8 WatchSource:0}: Error finding container bdcf4430ffda30b0ffa24fbb91f03c35c4b78da72e73571b9d4cb1640ef582a8: Status 404 returned error can't find the container with id bdcf4430ffda30b0ffa24fbb91f03c35c4b78da72e73571b9d4cb1640ef582a8 Jul 17 18:11:27.695927 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.695898 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696038 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.695955 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696038 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696014 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696060 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-serving-certs-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696108 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-metrics-client-ca\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696550 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696179 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696550 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696220 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-federate-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696550 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696261 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bqh\" (UniqueName: \"kubernetes.io/projected/0f535c45-1476-4127-bd45-6b33e94405f1-kube-api-access-74bqh\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.696704 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.696584 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c599bf6db-9h9cw"] Jul 17 18:11:27.697774 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.697445 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-serving-certs-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.698041 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.698016 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-metrics-client-ca\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.698828 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.698801 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.699569 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.699542 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-telemeter-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.700537 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:27.700289 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc171558a_c4c4_417d_bd00_9624f79cb9ae.slice/crio-a64bb2774f2f74889f31918804027d339a29da0a0deaa5e60dc8a7e5e6a3c328 WatchSource:0}: Error finding container a64bb2774f2f74889f31918804027d339a29da0a0deaa5e60dc8a7e5e6a3c328: Status 404 returned error can't find the container with id a64bb2774f2f74889f31918804027d339a29da0a0deaa5e60dc8a7e5e6a3c328 Jul 17 18:11:27.700537 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.700516 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.702589 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.701615 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-federate-client-tls\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.704143 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.704120 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bqh\" (UniqueName: \"kubernetes.io/projected/0f535c45-1476-4127-bd45-6b33e94405f1-kube-api-access-74bqh\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.705023 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.704972 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f535c45-1476-4127-bd45-6b33e94405f1-secret-telemeter-client\") pod \"telemeter-client-755bfb49b6-lzz5v\" (UID: \"0f535c45-1476-4127-bd45-6b33e94405f1\") " pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:27.791166 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:27.791134 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" Jul 17 18:11:28.041255 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.041198 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-755bfb49b6-lzz5v"] Jul 17 18:11:28.042818 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:28.042795 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f535c45_1476_4127_bd45_6b33e94405f1.slice/crio-7ec229d19e39ef0b844979aa1769c119a93a1c711e2fb2b0241b2a972756bebf WatchSource:0}: Error finding container 7ec229d19e39ef0b844979aa1769c119a93a1c711e2fb2b0241b2a972756bebf: Status 404 returned error can't find the container with id 7ec229d19e39ef0b844979aa1769c119a93a1c711e2fb2b0241b2a972756bebf Jul 17 18:11:28.359556 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.359373 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" event={"ID":"0f535c45-1476-4127-bd45-6b33e94405f1","Type":"ContainerStarted","Data":"7ec229d19e39ef0b844979aa1769c119a93a1c711e2fb2b0241b2a972756bebf"} Jul 17 18:11:28.361958 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.361929 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"be57883484d3bfdc1d13e2aa7ab879d32dd2bd072b419ea664ddc327045772a8"} Jul 17 18:11:28.362088 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.362014 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"0291692aa2889cf8153b639dfb4a60bcc76ca14d0c8ad240d13abef5619ba252"} Jul 17 18:11:28.362088 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.362059 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"e9d68aa59d1601e6b6d5ea3453ebf93887942825274970b722ed6e9b9af791e8"} Jul 17 18:11:28.363197 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.363169 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" event={"ID":"c171558a-c4c4-417d-bd00-9624f79cb9ae","Type":"ContainerStarted","Data":"a64bb2774f2f74889f31918804027d339a29da0a0deaa5e60dc8a7e5e6a3c328"} Jul 17 18:11:28.364445 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:28.364425 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" event={"ID":"b8af67c7-2a92-4f12-80a9-4663c3d6a19b","Type":"ContainerStarted","Data":"bdcf4430ffda30b0ffa24fbb91f03c35c4b78da72e73571b9d4cb1640ef582a8"} Jul 17 18:11:32.383745 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.383699 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" event={"ID":"0f535c45-1476-4127-bd45-6b33e94405f1","Type":"ContainerStarted","Data":"2ec7f55e0be6633252d85ebe8610054721a1d5f15b5ff6396428935c39cc7d10"} Jul 17 18:11:32.388052 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.388019 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"9361b26d6245b0eebc58d184181c242cf4e1276ac26335d7ee6245a122b36950"} Jul 17 18:11:32.388181 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.388099 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"116d9866e42a7ac21bb2a99a5cc3fb22544ab78c6f3b6524e6090a0b9924ffea"} Jul 17 18:11:32.388181 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.388143 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" event={"ID":"90ecdd24-7814-494b-8674-6ef55ae965e8","Type":"ContainerStarted","Data":"5ce121d89b2e04bd9815187d620713b0180bb533b1114e66217a34009df0fc0e"} Jul 17 18:11:32.388347 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.388329 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:32.390150 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.390122 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" event={"ID":"c171558a-c4c4-417d-bd00-9624f79cb9ae","Type":"ContainerStarted","Data":"94cb71db3c43403583f398406e6668a6901a047f5d33df1014a16da78cc84462"} Jul 17 18:11:32.391780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.391754 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" event={"ID":"b8af67c7-2a92-4f12-80a9-4663c3d6a19b","Type":"ContainerStarted","Data":"7c002668b8f182c085f58fdf34ad43dbc311387b6d0d3c7120cbcb917a4e1b64"} Jul 17 18:11:32.392003 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.391939 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:32.398035 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.398018 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" Jul 17 18:11:32.411108 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.410855 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" podStartSLOduration=2.676717005 podStartE2EDuration="8.410840308s" totalImagesPullingTime="5.734123303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:32.409752894 +0000 UTC m=+99.228524158" watchObservedRunningTime="2026-07-17 18:11:32.410840308 +0000 UTC m=+99.229611571" Jul 17 18:11:32.426501 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.426441 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" podStartSLOduration=2.710595577 podStartE2EDuration="6.426426763s" totalImagesPullingTime="3.715831186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:32.425718504 +0000 UTC m=+99.244489768" watchObservedRunningTime="2026-07-17 18:11:32.426426763 +0000 UTC m=+99.245198027" Jul 17 18:11:32.441832 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:32.441786 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-t4fdw" podStartSLOduration=2.702183234 podStartE2EDuration="6.441772021s" totalImagesPullingTime="3.739588787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:32.440768968 +0000 UTC m=+99.259540236" watchObservedRunningTime="2026-07-17 18:11:32.441772021 +0000 UTC m=+99.260543286" Jul 17 18:11:33.399103 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:33.399055 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" event={"ID":"0f535c45-1476-4127-bd45-6b33e94405f1","Type":"ContainerStarted","Data":"9354c448d0255d24c401a35ab7f81fe2fd0bcad4333694493cba448d99ef15c4"} Jul 17 18:11:33.406745 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:33.406672 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-556b84bb4f-6zjqt" Jul 17 18:11:34.405904 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:34.405869 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" event={"ID":"0f535c45-1476-4127-bd45-6b33e94405f1","Type":"ContainerStarted","Data":"ac2f1588249bc11a2fd9380b30596a3322f542202e5e4baa5e28649a8e6bac9f"} Jul 17 18:11:34.426036 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:34.425967 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-755bfb49b6-lzz5v" podStartSLOduration=2.5067133 podStartE2EDuration="7.425947639s" totalImagesPullingTime="4.919234339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:34.424850717 +0000 UTC m=+101.243621996" watchObservedRunningTime="2026-07-17 18:11:34.425947639 +0000 UTC m=+101.244718898" Jul 17 18:11:35.330928 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.330898 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:11:35.357956 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.357929 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:11:35.358115 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.358043 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.364581 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.364285 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 18:11:35.474419 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474376 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474428 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474460 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp77j\" (UniqueName: \"kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474507 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474554 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474656 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.474848 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.474726 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.575624 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.575584 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.575801 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.575647 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.575882 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.575807 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.575947 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.575880 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576042 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.575945 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp77j\" (UniqueName: \"kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.576065 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.576095 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576652 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.576622 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576826 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.576782 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.576933 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.576857 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.577273 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.577247 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.579503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.579483 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.579616 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.579596 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.582878 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.582831 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp77j\" (UniqueName: \"kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j\") pod \"console-56ccb7667c-gk4vr\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.667264 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.666961 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:35.837064 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:35.836965 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:11:35.841449 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:11:35.841405 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabde9d79_bb2a_430f_88da_fa7347c5102a.slice/crio-1bee831ae32c8498c5276d1b76af8d295139fbe65465a0a8bb975cfe617c1809 WatchSource:0}: Error finding container 1bee831ae32c8498c5276d1b76af8d295139fbe65465a0a8bb975cfe617c1809: Status 404 returned error can't find the container with id 1bee831ae32c8498c5276d1b76af8d295139fbe65465a0a8bb975cfe617c1809 Jul 17 18:11:36.416505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:36.416469 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccb7667c-gk4vr" event={"ID":"abde9d79-bb2a-430f-88da-fa7347c5102a","Type":"ContainerStarted","Data":"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3"} Jul 17 18:11:36.416687 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:36.416525 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccb7667c-gk4vr" event={"ID":"abde9d79-bb2a-430f-88da-fa7347c5102a","Type":"ContainerStarted","Data":"1bee831ae32c8498c5276d1b76af8d295139fbe65465a0a8bb975cfe617c1809"} Jul 17 18:11:36.438506 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:36.438458 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56ccb7667c-gk4vr" podStartSLOduration=1.438442661 podStartE2EDuration="1.438442661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:36.436794994 +0000 UTC m=+103.255566284" watchObservedRunningTime="2026-07-17 18:11:36.438442661 +0000 UTC m=+103.257213925" Jul 17 18:11:39.252026 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:39.251978 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9lcq9" Jul 17 18:11:45.325263 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.325201 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-698bff8fb7-chh4c" podUID="5379773a-ba43-468c-bc21-24555bafaa31" containerName="console" containerID="cri-o://42753cd077bdb47f1fd97b7534bce80064e25b8f459532817988fba48349984c" gracePeriod=15 Jul 17 18:11:45.453165 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.453135 2587 generic.go:356] "Generic (PLEG): container finished" podID="5379773a-ba43-468c-bc21-24555bafaa31" containerID="42753cd077bdb47f1fd97b7534bce80064e25b8f459532817988fba48349984c" exitCode=0 Jul 17 18:11:45.453307 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.453209 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698bff8fb7-chh4c" event={"ID":"5379773a-ba43-468c-bc21-24555bafaa31","Type":"ContainerDied","Data":"42753cd077bdb47f1fd97b7534bce80064e25b8f459532817988fba48349984c"} Jul 17 18:11:45.590872 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.590812 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:45.661740 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.661708 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7vhj\" (UniqueName: \"kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.661923 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.661753 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.661923 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.661782 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.661923 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.661817 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.662117 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.661930 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.662117 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.662024 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca\") pod \"5379773a-ba43-468c-bc21-24555bafaa31\" (UID: \"5379773a-ba43-468c-bc21-24555bafaa31\") " Jul 17 18:11:45.662328 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.662291 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:11:45.662452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.662295 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config" (OuterVolumeSpecName: "console-config") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:11:45.662581 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.662551 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca" (OuterVolumeSpecName: "service-ca") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:11:45.664241 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.664215 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:11:45.664315 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.664215 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj" (OuterVolumeSpecName: "kube-api-access-r7vhj") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "kube-api-access-r7vhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:11:45.664315 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.664281 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5379773a-ba43-468c-bc21-24555bafaa31" (UID: "5379773a-ba43-468c-bc21-24555bafaa31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:11:45.668476 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.668333 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:45.668476 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.668381 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:45.673110 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.673094 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:45.762865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762811 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vhj\" (UniqueName: \"kubernetes.io/projected/5379773a-ba43-468c-bc21-24555bafaa31-kube-api-access-r7vhj\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:45.762865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762840 2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:45.762865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762856 2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-oauth-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:45.762865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762870 2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-console-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:45.763217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762884 2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5379773a-ba43-468c-bc21-24555bafaa31-console-oauth-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:45.763217 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:45.762897 2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5379773a-ba43-468c-bc21-24555bafaa31-service-ca\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:11:46.458640 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.458609 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698bff8fb7-chh4c" Jul 17 18:11:46.459237 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.458649 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698bff8fb7-chh4c" event={"ID":"5379773a-ba43-468c-bc21-24555bafaa31","Type":"ContainerDied","Data":"5f50831c2abf36f4ac7f96b8aa195a29c4e7bc9acfc3233d7798668fb1e402fe"} Jul 17 18:11:46.459237 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.458706 2587 scope.go:117] "RemoveContainer" containerID="42753cd077bdb47f1fd97b7534bce80064e25b8f459532817988fba48349984c" Jul 17 18:11:46.463403 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.463384 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:11:46.472500 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.472477 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:46.476498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.476476 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-698bff8fb7-chh4c"] Jul 17 18:11:46.956745 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.956711 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:46.956919 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:46.956772 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:11:47.739131 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:11:47.739091 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5379773a-ba43-468c-bc21-24555bafaa31" path="/var/lib/kubelet/pods/5379773a-ba43-468c-bc21-24555bafaa31/volumes" Jul 17 18:12:06.962657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:06.962625 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:12:06.966459 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:06.966430 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c599bf6db-9h9cw" Jul 17 18:12:44.109712 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.109680 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 18:12:44.110211 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.110018 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5379773a-ba43-468c-bc21-24555bafaa31" containerName="console" Jul 17 18:12:44.110211 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.110030 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5379773a-ba43-468c-bc21-24555bafaa31" containerName="console" Jul 17 18:12:44.110211 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.110082 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5379773a-ba43-468c-bc21-24555bafaa31" containerName="console" Jul 17 18:12:44.114775 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.114754 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.116958 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.116907 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 18:12:44.116958 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.116921 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 18:12:44.117174 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.116972 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 18:12:44.117174 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117049 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wdnjn\"" Jul 17 18:12:44.117267 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117187 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 18:12:44.117309 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117263 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 18:12:44.117450 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117435 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 18:12:44.117498 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117449 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 18:12:44.117722 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.117705 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 18:12:44.123815 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.123793 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 18:12:44.131637 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.131615 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 18:12:44.142113 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142090 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-tls-assets\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142127 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-config-out\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142149 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142196 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142227 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5gf\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-kube-api-access-vg5gf\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142257 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-web-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142274 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142329 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142295 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142358 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142375 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142390 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142499 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.142542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.142537 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-config-volume\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243285 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243252 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243285 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243288 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243306 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243340 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243356 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-config-volume\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243376 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-tls-assets\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243404 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-config-out\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243419 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243437 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243458 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5gf\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-kube-api-access-vg5gf\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243478 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-web-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243493 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.243551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243524 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.244166 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.243755 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.244361 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.244336 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.244713 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.244685 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067c77b2-f82c-47a1-af81-d775c603f802-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.246751 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.246705 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/067c77b2-f82c-47a1-af81-d775c603f802-config-out\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.246866 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.246829 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-web-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247188 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247161 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247295 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247187 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-config-volume\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247295 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247189 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247295 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247242 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-tls-assets\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247468 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247302 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.247526 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.247503 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.248222 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.248202 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/067c77b2-f82c-47a1-af81-d775c603f802-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.252612 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.252586 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5gf\" (UniqueName: \"kubernetes.io/projected/067c77b2-f82c-47a1-af81-d775c603f802-kube-api-access-vg5gf\") pod \"alertmanager-main-0\" (UID: \"067c77b2-f82c-47a1-af81-d775c603f802\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.358450 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.358418 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:12:44.361543 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.361502 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.373455 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.372935 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:12:44.425288 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.425251 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 18:12:44.445687 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445652 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.445832 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445703 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.445832 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445733 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.445924 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445829 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.445924 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445883 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.446006 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445933 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.446006 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.445959 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9x9\" (UniqueName: \"kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547247 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547299 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547322 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547342 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9x9\" (UniqueName: \"kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547415 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547439 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.547711 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.547462 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.548090 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.548060 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.548234 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.548113 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.548234 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.548132 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.548348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.548247 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.550044 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.550017 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.550128 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.550105 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.554383 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.554362 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9x9\" (UniqueName: \"kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9\") pod \"console-86cc6d594-jwncj\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.562231 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.562209 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 18:12:44.565158 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:12:44.565132 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067c77b2_f82c_47a1_af81_d775c603f802.slice/crio-02eb74125ba03617ff57c299dd49aa6ded775627a0dfbb93533c32c512ae5493 WatchSource:0}: Error finding container 02eb74125ba03617ff57c299dd49aa6ded775627a0dfbb93533c32c512ae5493: Status 404 returned error can't find the container with id 02eb74125ba03617ff57c299dd49aa6ded775627a0dfbb93533c32c512ae5493 Jul 17 18:12:44.671123 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.671087 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:44.682551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.682525 2587 generic.go:356] "Generic (PLEG): container finished" podID="067c77b2-f82c-47a1-af81-d775c603f802" containerID="1f1a99fbeaf6bd5910bc2e65a844e1a035c5996c936b8256d62500ca40faac6b" exitCode=0 Jul 17 18:12:44.682652 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.682561 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerDied","Data":"1f1a99fbeaf6bd5910bc2e65a844e1a035c5996c936b8256d62500ca40faac6b"} Jul 17 18:12:44.682652 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.682610 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"02eb74125ba03617ff57c299dd49aa6ded775627a0dfbb93533c32c512ae5493"} Jul 17 18:12:44.799275 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:44.799247 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:12:44.801135 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:12:44.801110 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8776472_681b_45ea_a6ca_2d6893f080dc.slice/crio-79d614ab33d13015fd5a344a92c7fa11968e9fd5feea42da6cc8a4838ebd2199 WatchSource:0}: Error finding container 79d614ab33d13015fd5a344a92c7fa11968e9fd5feea42da6cc8a4838ebd2199: Status 404 returned error can't find the container with id 79d614ab33d13015fd5a344a92c7fa11968e9fd5feea42da6cc8a4838ebd2199 Jul 17 18:12:45.689315 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:45.689231 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc6d594-jwncj" event={"ID":"b8776472-681b-45ea-a6ca-2d6893f080dc","Type":"ContainerStarted","Data":"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019"} Jul 17 18:12:45.689315 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:45.689282 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc6d594-jwncj" event={"ID":"b8776472-681b-45ea-a6ca-2d6893f080dc","Type":"ContainerStarted","Data":"79d614ab33d13015fd5a344a92c7fa11968e9fd5feea42da6cc8a4838ebd2199"} Jul 17 18:12:45.705584 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:45.705530 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86cc6d594-jwncj" podStartSLOduration=1.705507498 podStartE2EDuration="1.705507498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:12:45.704898398 +0000 UTC m=+172.523669671" watchObservedRunningTime="2026-07-17 18:12:45.705507498 +0000 UTC m=+172.524278763" Jul 17 18:12:46.697437 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697351 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"13661622183caca469f0f714ad84b00124cacb63ca16b41b77b5028328a4c7a4"} Jul 17 18:12:46.697437 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697427 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"fdaed52c59b06eed8b6ed59e76920b48b26a59b1ccb926ccf707b82a9dd402dc"} Jul 17 18:12:46.697833 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697452 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"e72f85687fb30021a62d748845952f89e264e80f129144aadea5a677124e6ac4"} Jul 17 18:12:46.697833 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697476 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"e1db058480138eafa76f53fee8ded1f78083d9b719c8ae93d58722458c91ff7b"} Jul 17 18:12:46.697833 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697500 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"988184278ac3dc50db43a4c685110a5d39929c01e63b759435da0c839288acb9"} Jul 17 18:12:46.697833 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.697526 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"067c77b2-f82c-47a1-af81-d775c603f802","Type":"ContainerStarted","Data":"5f4a70bbc83217fa4b25b6cd091d818eeab60e2be754c3814c796c07a431ab65"} Jul 17 18:12:46.727697 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:46.727637 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.376674671 podStartE2EDuration="2.72761702s" totalImagesPullingTime="1.33035011s" totalInitContainerRuntime="20.592239ms" isStatefulPod=false podCreationTimestamp="2026-07-17 18:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:12:46.725165582 +0000 UTC m=+173.543936855" watchObservedRunningTime="2026-07-17 18:12:46.72761702 +0000 UTC m=+173.546388286" Jul 17 18:12:54.671539 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:54.671498 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:54.671938 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:54.671561 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:54.676302 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:54.676280 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:54.733868 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:54.733842 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:12:54.769579 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:12:54.769545 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:13:19.790223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:19.790156 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56ccb7667c-gk4vr" podUID="abde9d79-bb2a-430f-88da-fa7347c5102a" containerName="console" containerID="cri-o://9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3" gracePeriod=15 Jul 17 18:13:20.039129 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.039105 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:13:20.154084 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154007 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154084 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154060 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154090 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp77j\" (UniqueName: \"kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154109 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154127 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154181 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154217 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config\") pod \"abde9d79-bb2a-430f-88da-fa7347c5102a\" (UID: \"abde9d79-bb2a-430f-88da-fa7347c5102a\") " Jul 17 18:13:20.154546 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154435 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:20.154611 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154548 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:20.154713 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154591 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca" (OuterVolumeSpecName: "service-ca") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:20.154782 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.154746 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config" (OuterVolumeSpecName: "console-config") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:13:20.156412 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.156385 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:13:20.156476 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.156392 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j" (OuterVolumeSpecName: "kube-api-access-hp77j") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "kube-api-access-hp77j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:13:20.156476 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.156396 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "abde9d79-bb2a-430f-88da-fa7347c5102a" (UID: "abde9d79-bb2a-430f-88da-fa7347c5102a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:13:20.255347 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255317 2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-oauth-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255347 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255345 2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-console-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255347 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255357 2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-trusted-ca-bundle\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255579 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255369 2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-service-ca\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255579 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255383 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp77j\" (UniqueName: \"kubernetes.io/projected/abde9d79-bb2a-430f-88da-fa7347c5102a-kube-api-access-hp77j\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255579 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255393 2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abde9d79-bb2a-430f-88da-fa7347c5102a-oauth-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.255579 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.255406 2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abde9d79-bb2a-430f-88da-fa7347c5102a-console-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:13:20.834076 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.834043 2587 generic.go:356] "Generic (PLEG): container finished" podID="abde9d79-bb2a-430f-88da-fa7347c5102a" containerID="9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3" exitCode=0 Jul 17 18:13:20.834543 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.834088 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56ccb7667c-gk4vr" Jul 17 18:13:20.834543 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.834097 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccb7667c-gk4vr" event={"ID":"abde9d79-bb2a-430f-88da-fa7347c5102a","Type":"ContainerDied","Data":"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3"} Jul 17 18:13:20.834543 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.834147 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56ccb7667c-gk4vr" event={"ID":"abde9d79-bb2a-430f-88da-fa7347c5102a","Type":"ContainerDied","Data":"1bee831ae32c8498c5276d1b76af8d295139fbe65465a0a8bb975cfe617c1809"} Jul 17 18:13:20.834543 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.834174 2587 scope.go:117] "RemoveContainer" containerID="9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3" Jul 17 18:13:20.842532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.842517 2587 scope.go:117] "RemoveContainer" containerID="9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3" Jul 17 18:13:20.842801 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:13:20.842779 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3\": container with ID starting with 9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3 not found: ID does not exist" containerID="9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3" Jul 17 18:13:20.842843 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.842809 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3"} err="failed to get container status \"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3\": rpc error: code = NotFound desc = could not find container \"9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3\": container with ID starting with 9804ff019ecc4189decd61b779dc4c2dbe4b0d0d3adc083e612082cebcbafec3 not found: ID does not exist" Jul 17 18:13:20.853006 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.852969 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:13:20.857943 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:20.857919 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56ccb7667c-gk4vr"] Jul 17 18:13:21.739569 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:13:21.739536 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abde9d79-bb2a-430f-88da-fa7347c5102a" path="/var/lib/kubelet/pods/abde9d79-bb2a-430f-88da-fa7347c5102a/volumes" Jul 17 18:14:15.420171 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.420074 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mdq7g"] Jul 17 18:14:15.420567 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.420406 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abde9d79-bb2a-430f-88da-fa7347c5102a" containerName="console" Jul 17 18:14:15.420567 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.420418 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="abde9d79-bb2a-430f-88da-fa7347c5102a" containerName="console" Jul 17 18:14:15.420567 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.420482 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="abde9d79-bb2a-430f-88da-fa7347c5102a" containerName="console" Jul 17 18:14:15.423212 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.423196 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.425236 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.425212 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 18:14:15.425665 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.425648 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-glklm\"" Jul 17 18:14:15.425719 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.425662 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 18:14:15.434032 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.434009 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mdq7g"] Jul 17 18:14:15.508811 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.508775 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.508974 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.508814 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzql4\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-kube-api-access-jzql4\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.609873 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.609837 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.610054 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.609883 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzql4\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-kube-api-access-jzql4\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.623074 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.623048 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.623278 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.623260 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzql4\" (UniqueName: \"kubernetes.io/projected/eb338e78-d514-4a67-b145-6a1564ea89c2-kube-api-access-jzql4\") pod \"cert-manager-webhook-6486ffd997-mdq7g\" (UID: \"eb338e78-d514-4a67-b145-6a1564ea89c2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.731400 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.731330 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:15.856235 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:15.856212 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mdq7g"] Jul 17 18:14:15.858311 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:14:15.858288 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb338e78_d514_4a67_b145_6a1564ea89c2.slice/crio-b7a84a6caad93cbd6a6c9605e900888f89e84cecbd4403e19be9524a6b5d9ae2 WatchSource:0}: Error finding container b7a84a6caad93cbd6a6c9605e900888f89e84cecbd4403e19be9524a6b5d9ae2: Status 404 returned error can't find the container with id b7a84a6caad93cbd6a6c9605e900888f89e84cecbd4403e19be9524a6b5d9ae2 Jul 17 18:14:16.051224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:16.051189 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" event={"ID":"eb338e78-d514-4a67-b145-6a1564ea89c2","Type":"ContainerStarted","Data":"b7a84a6caad93cbd6a6c9605e900888f89e84cecbd4403e19be9524a6b5d9ae2"} Jul 17 18:14:20.071159 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:20.071120 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" event={"ID":"eb338e78-d514-4a67-b145-6a1564ea89c2","Type":"ContainerStarted","Data":"f87783f10bfc99734f15d2b82611bbcb9c5ac7cc64b97e060ee6240bb94a503d"} Jul 17 18:14:20.071528 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:20.071211 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:20.090764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:20.090719 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" podStartSLOduration=1.795472315 podStartE2EDuration="5.090703104s" totalImagesPullingTime="3.295230789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:20.089589062 +0000 UTC m=+266.908360326" watchObservedRunningTime="2026-07-17 18:14:20.090703104 +0000 UTC m=+266.909474366" Jul 17 18:14:26.077031 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:26.076976 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mdq7g" Jul 17 18:14:48.865890 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.865859 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2"] Jul 17 18:14:48.869232 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.869213 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:48.872656 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.872626 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 18:14:48.872851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.872675 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 18:14:48.872851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.872708 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wn5nw\"" Jul 17 18:14:48.872851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.872680 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 18:14:48.873055 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.872959 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 18:14:48.873055 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.873000 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:14:48.891017 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.890976 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2"] Jul 17 18:14:48.985671 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.985625 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b5adc00-dc8b-4d16-bd02-0cfefa904820-manager-config\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:48.985671 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.985670 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5bc\" (UniqueName: \"kubernetes.io/projected/7b5adc00-dc8b-4d16-bd02-0cfefa904820-kube-api-access-9g5bc\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:48.985875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.985728 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:48.985875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:48.985760 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-metrics-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.086872 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.086834 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b5adc00-dc8b-4d16-bd02-0cfefa904820-manager-config\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.086872 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.086876 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5bc\" (UniqueName: \"kubernetes.io/projected/7b5adc00-dc8b-4d16-bd02-0cfefa904820-kube-api-access-9g5bc\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.087065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.087019 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.087065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.087056 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-metrics-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.087650 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.087626 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b5adc00-dc8b-4d16-bd02-0cfefa904820-manager-config\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.089666 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.089646 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.089737 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.089719 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b5adc00-dc8b-4d16-bd02-0cfefa904820-metrics-cert\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.101691 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.101659 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5bc\" (UniqueName: \"kubernetes.io/projected/7b5adc00-dc8b-4d16-bd02-0cfefa904820-kube-api-access-9g5bc\") pod \"lws-controller-manager-7cf59f64f5-nrkl2\" (UID: \"7b5adc00-dc8b-4d16-bd02-0cfefa904820\") " pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.178294 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.178196 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:49.315892 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.315864 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2"] Jul 17 18:14:49.318530 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:14:49.318498 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5adc00_dc8b_4d16_bd02_0cfefa904820.slice/crio-2cf9aff94f417d58219dc3cf256940d59bfe1464e4a6c1dc208a9ca6cb6cd0ff WatchSource:0}: Error finding container 2cf9aff94f417d58219dc3cf256940d59bfe1464e4a6c1dc208a9ca6cb6cd0ff: Status 404 returned error can't find the container with id 2cf9aff94f417d58219dc3cf256940d59bfe1464e4a6c1dc208a9ca6cb6cd0ff Jul 17 18:14:49.560367 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.560329 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h"] Jul 17 18:14:49.564908 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.564883 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.567209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.567186 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 18:14:49.567348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.567275 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 18:14:49.567417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.567357 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 18:14:49.567695 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.567677 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z9sgw\"" Jul 17 18:14:49.568440 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.568426 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 18:14:49.581474 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.581450 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h"] Jul 17 18:14:49.692830 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.692789 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.693053 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.692845 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rm7\" (UniqueName: \"kubernetes.io/projected/7028b7bf-1c02-4809-8879-ef8088f88e68-kube-api-access-v7rm7\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.693053 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.692920 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.693053 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.692951 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.693053 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.693039 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.793755 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.793720 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.793755 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.793753 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.794031 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.793794 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.794031 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.793855 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.794031 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.793883 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rm7\" (UniqueName: \"kubernetes.io/projected/7028b7bf-1c02-4809-8879-ef8088f88e68-kube-api-access-v7rm7\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.794186 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.794149 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-manifests\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.796140 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.796116 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7028b7bf-1c02-4809-8879-ef8088f88e68-tmp\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.796358 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.796341 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-webhook-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.796398 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.796363 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7028b7bf-1c02-4809-8879-ef8088f88e68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.801224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.801196 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rm7\" (UniqueName: \"kubernetes.io/projected/7028b7bf-1c02-4809-8879-ef8088f88e68-kube-api-access-v7rm7\") pod \"opendatahub-operator-controller-manager-9846c54f8-99g9h\" (UID: \"7028b7bf-1c02-4809-8879-ef8088f88e68\") " pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:49.876253 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:49.876178 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:50.017120 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:50.016804 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h"] Jul 17 18:14:50.020027 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:14:50.019939 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7028b7bf_1c02_4809_8879_ef8088f88e68.slice/crio-d353b3e8f8458bd3c611afbeab4f07d39cccd92cbff30af470384e8737f874ab WatchSource:0}: Error finding container d353b3e8f8458bd3c611afbeab4f07d39cccd92cbff30af470384e8737f874ab: Status 404 returned error can't find the container with id d353b3e8f8458bd3c611afbeab4f07d39cccd92cbff30af470384e8737f874ab Jul 17 18:14:50.189849 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:50.189730 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" event={"ID":"7028b7bf-1c02-4809-8879-ef8088f88e68","Type":"ContainerStarted","Data":"d353b3e8f8458bd3c611afbeab4f07d39cccd92cbff30af470384e8737f874ab"} Jul 17 18:14:50.191621 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:50.191587 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" event={"ID":"7b5adc00-dc8b-4d16-bd02-0cfefa904820","Type":"ContainerStarted","Data":"2cf9aff94f417d58219dc3cf256940d59bfe1464e4a6c1dc208a9ca6cb6cd0ff"} Jul 17 18:14:52.208911 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:52.208872 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" event={"ID":"7b5adc00-dc8b-4d16-bd02-0cfefa904820","Type":"ContainerStarted","Data":"887e2a602efe3c954820630e7d4f49e56779a5fcc16a7f1673653a049a8857c5"} Jul 17 18:14:52.209397 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:52.209012 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:14:53.964612 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:53.964581 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:14:53.966844 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:53.964834 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:14:53.967320 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:53.967305 2587 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 18:14:54.218913 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:54.218825 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" event={"ID":"7028b7bf-1c02-4809-8879-ef8088f88e68","Type":"ContainerStarted","Data":"21711cfe4f67efcea6b471d629b9e98f996cbba58ff9d9fccebe5f0a05b9a072"} Jul 17 18:14:54.241792 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:54.241724 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" podStartSLOduration=3.903618146 podStartE2EDuration="6.241705965s" totalImagesPullingTime="2.338087819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:52.224059943 +0000 UTC m=+299.042831207" watchObservedRunningTime="2026-07-17 18:14:54.241705965 +0000 UTC m=+301.060477228" Jul 17 18:14:55.224603 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:55.224565 2587 generic.go:356] "Generic (PLEG): container finished" podID="7028b7bf-1c02-4809-8879-ef8088f88e68" containerID="21711cfe4f67efcea6b471d629b9e98f996cbba58ff9d9fccebe5f0a05b9a072" exitCode=0 Jul 17 18:14:55.224962 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:55.224650 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" event={"ID":"7028b7bf-1c02-4809-8879-ef8088f88e68","Type":"ContainerDied","Data":"21711cfe4f67efcea6b471d629b9e98f996cbba58ff9d9fccebe5f0a05b9a072"} Jul 17 18:14:55.225140 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:55.225125 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:14:56.231197 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:56.231164 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" event={"ID":"7028b7bf-1c02-4809-8879-ef8088f88e68","Type":"ContainerStarted","Data":"d842bb17c93e45520aeecc6e66637c6baaed90f661b0ef83624cfb4c1c163e9c"} Jul 17 18:14:56.231596 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:56.231256 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:14:56.250806 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:14:56.250632 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" podStartSLOduration=1.7507639510000002 podStartE2EDuration="7.250616272s" totalImagesPullingTime="4.42785886s" totalInitContainerRuntime="1.071993461s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:56.249327557 +0000 UTC m=+303.068098823" watchObservedRunningTime="2026-07-17 18:14:56.250616272 +0000 UTC m=+303.069387537" Jul 17 18:15:03.216070 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:03.216037 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cf59f64f5-nrkl2" Jul 17 18:15:07.237785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:07.237754 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9846c54f8-99g9h" Jul 17 18:15:09.722212 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.722173 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq"] Jul 17 18:15:09.732178 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.732157 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.735240 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.735212 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 18:15:09.735569 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.735552 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 18:15:09.735818 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.735803 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tz8nf\"" Jul 17 18:15:09.736098 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.736079 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 18:15:09.736387 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.736372 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 18:15:09.743483 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.743462 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq"] Jul 17 18:15:09.870964 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.870922 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f83132-f55f-42d9-91d0-e95d5e4af89e-tls-certs\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.871191 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.871163 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00f83132-f55f-42d9-91d0-e95d5e4af89e-tmp\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.871267 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.871247 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47pqh\" (UniqueName: \"kubernetes.io/projected/00f83132-f55f-42d9-91d0-e95d5e4af89e-kube-api-access-47pqh\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.972864 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.972770 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f83132-f55f-42d9-91d0-e95d5e4af89e-tls-certs\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.972864 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.972826 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00f83132-f55f-42d9-91d0-e95d5e4af89e-tmp\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.972864 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.972862 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47pqh\" (UniqueName: \"kubernetes.io/projected/00f83132-f55f-42d9-91d0-e95d5e4af89e-kube-api-access-47pqh\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.975120 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.975093 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00f83132-f55f-42d9-91d0-e95d5e4af89e-tmp\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.975575 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.975551 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f83132-f55f-42d9-91d0-e95d5e4af89e-tls-certs\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:09.990244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:09.990203 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47pqh\" (UniqueName: \"kubernetes.io/projected/00f83132-f55f-42d9-91d0-e95d5e4af89e-kube-api-access-47pqh\") pod \"kube-auth-proxy-6b79874b6-vwtnq\" (UID: \"00f83132-f55f-42d9-91d0-e95d5e4af89e\") " pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:10.043359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:10.043326 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" Jul 17 18:15:10.192517 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:10.192488 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq"] Jul 17 18:15:10.194649 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:15:10.194619 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f83132_f55f_42d9_91d0_e95d5e4af89e.slice/crio-751888a7d59927de8681d085bf66510656f5d81084c4d7ea6f091739ffb11e0d WatchSource:0}: Error finding container 751888a7d59927de8681d085bf66510656f5d81084c4d7ea6f091739ffb11e0d: Status 404 returned error can't find the container with id 751888a7d59927de8681d085bf66510656f5d81084c4d7ea6f091739ffb11e0d Jul 17 18:15:10.297438 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:10.297401 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" event={"ID":"00f83132-f55f-42d9-91d0-e95d5e4af89e","Type":"ContainerStarted","Data":"751888a7d59927de8681d085bf66510656f5d81084c4d7ea6f091739ffb11e0d"} Jul 17 18:15:14.317901 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:14.317860 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" event={"ID":"00f83132-f55f-42d9-91d0-e95d5e4af89e","Type":"ContainerStarted","Data":"bbd6eba7d9581c0403df00bbedc199812c655404cac407585996fca491235c20"} Jul 17 18:15:14.332582 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:15:14.332518 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b79874b6-vwtnq" podStartSLOduration=2.029454299 podStartE2EDuration="5.332497614s" totalImagesPullingTime="3.303043315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:14.331876014 +0000 UTC m=+321.150647282" watchObservedRunningTime="2026-07-17 18:15:14.332497614 +0000 UTC m=+321.151268878" Jul 17 18:16:50.294636 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.294596 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fbccc4567-xc8gm"] Jul 17 18:16:50.298218 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.298193 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.312603 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.312577 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbccc4567-xc8gm"] Jul 17 18:16:50.454447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454406 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454451 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-trusted-ca-bundle\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454470 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-oauth-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454660 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454569 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-service-ca\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454666 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tjn\" (UniqueName: \"kubernetes.io/projected/60c1feac-96bb-475a-9c2f-d89b0bbdef99-kube-api-access-j6tjn\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454694 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.454729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.454715 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-oauth-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.555735 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555645 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tjn\" (UniqueName: \"kubernetes.io/projected/60c1feac-96bb-475a-9c2f-d89b0bbdef99-kube-api-access-j6tjn\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.555735 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555700 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.555900 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555807 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-oauth-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.555900 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555873 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555923 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-trusted-ca-bundle\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.555946 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-oauth-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.556024 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-service-ca\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.556688 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556836 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.556726 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-service-ca\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556836 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.556769 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-oauth-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.556836 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.556782 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60c1feac-96bb-475a-9c2f-d89b0bbdef99-trusted-ca-bundle\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.558294 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.558268 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-serving-cert\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.558406 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.558298 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60c1feac-96bb-475a-9c2f-d89b0bbdef99-console-oauth-config\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.569574 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.569552 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tjn\" (UniqueName: \"kubernetes.io/projected/60c1feac-96bb-475a-9c2f-d89b0bbdef99-kube-api-access-j6tjn\") pod \"console-5fbccc4567-xc8gm\" (UID: \"60c1feac-96bb-475a-9c2f-d89b0bbdef99\") " pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.607647 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.607593 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:16:50.750673 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:50.750571 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbccc4567-xc8gm"] Jul 17 18:16:50.753449 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:16:50.753424 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c1feac_96bb_475a_9c2f_d89b0bbdef99.slice/crio-78b9d32c6368ec913bf38333bbad67daa2567ac529b73dda25d6e98373f9b653 WatchSource:0}: Error finding container 78b9d32c6368ec913bf38333bbad67daa2567ac529b73dda25d6e98373f9b653: Status 404 returned error can't find the container with id 78b9d32c6368ec913bf38333bbad67daa2567ac529b73dda25d6e98373f9b653 Jul 17 18:16:51.746194 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:51.746150 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbccc4567-xc8gm" event={"ID":"60c1feac-96bb-475a-9c2f-d89b0bbdef99","Type":"ContainerStarted","Data":"64ee9c6311bb548e79f642001948eb138c7cf2526aeafa3fea590377253e17a1"} Jul 17 18:16:51.746572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:51.746205 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbccc4567-xc8gm" event={"ID":"60c1feac-96bb-475a-9c2f-d89b0bbdef99","Type":"ContainerStarted","Data":"78b9d32c6368ec913bf38333bbad67daa2567ac529b73dda25d6e98373f9b653"} Jul 17 18:16:51.779724 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:51.779670 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fbccc4567-xc8gm" podStartSLOduration=1.7796528600000001 podStartE2EDuration="1.77965286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:51.777919597 +0000 UTC m=+418.596690862" watchObservedRunningTime="2026-07-17 18:16:51.77965286 +0000 UTC m=+418.598424125" Jul 17 18:16:55.499383 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.499349 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75"] Jul 17 18:16:55.502823 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.502807 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.505427 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.505401 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zgbg7\"" Jul 17 18:16:55.505536 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.505516 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 17 18:16:55.506237 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.506220 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 18:16:55.506445 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.506430 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 17 18:16:55.506530 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.506436 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 18:16:55.519721 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.519700 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75"] Jul 17 18:16:55.604583 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.604549 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggx2\" (UniqueName: \"kubernetes.io/projected/76efa28c-2bd9-465f-9afd-40272685ebaf-kube-api-access-nggx2\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.604771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.604594 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76efa28c-2bd9-465f-9afd-40272685ebaf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.604771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.604688 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.705542 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.705507 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggx2\" (UniqueName: \"kubernetes.io/projected/76efa28c-2bd9-465f-9afd-40272685ebaf-kube-api-access-nggx2\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.705744 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.705553 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76efa28c-2bd9-465f-9afd-40272685ebaf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.705744 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.705614 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.705854 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:16:55.705766 2587 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 17 18:16:55.705854 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:16:55.705847 2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert podName:76efa28c-2bd9-465f-9afd-40272685ebaf nodeName:}" failed. No retries permitted until 2026-07-17 18:16:56.205824394 +0000 UTC m=+423.024595653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-wqh75" (UID: "76efa28c-2bd9-465f-9afd-40272685ebaf") : secret "plugin-serving-cert" not found Jul 17 18:16:55.706356 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.706334 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76efa28c-2bd9-465f-9afd-40272685ebaf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:55.717840 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:55.717810 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggx2\" (UniqueName: \"kubernetes.io/projected/76efa28c-2bd9-465f-9afd-40272685ebaf-kube-api-access-nggx2\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:56.211593 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:56.211551 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:56.214201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:56.214172 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76efa28c-2bd9-465f-9afd-40272685ebaf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wqh75\" (UID: \"76efa28c-2bd9-465f-9afd-40272685ebaf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:56.412605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:56.412566 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" Jul 17 18:16:56.543881 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:56.543711 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75"] Jul 17 18:16:56.546369 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:16:56.546344 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76efa28c_2bd9_465f_9afd_40272685ebaf.slice/crio-d468aab812184e070959b0357f489c3b3baabd0377f395ab0a5f152c930927d7 WatchSource:0}: Error finding container d468aab812184e070959b0357f489c3b3baabd0377f395ab0a5f152c930927d7: Status 404 returned error can't find the container with id d468aab812184e070959b0357f489c3b3baabd0377f395ab0a5f152c930927d7 Jul 17 18:16:56.768777 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:16:56.768741 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" event={"ID":"76efa28c-2bd9-465f-9afd-40272685ebaf","Type":"ContainerStarted","Data":"d468aab812184e070959b0357f489c3b3baabd0377f395ab0a5f152c930927d7"} Jul 17 18:17:00.608639 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:00.608602 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:17:00.609144 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:00.608672 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:17:00.614032 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:00.614005 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:17:00.797714 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:00.797680 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fbccc4567-xc8gm" Jul 17 18:17:00.865465 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:00.864657 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:17:18.890881 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:18.890841 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" event={"ID":"76efa28c-2bd9-465f-9afd-40272685ebaf","Type":"ContainerStarted","Data":"19bccdd163a184cffab181be669693067d9ee2d5c0146fcbd445be3cfb439205"} Jul 17 18:17:18.918752 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:18.918636 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wqh75" podStartSLOduration=1.77607025 podStartE2EDuration="23.918616846s" totalImagesPullingTime="22.142546596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:18.916101556 +0000 UTC m=+445.734872820" watchObservedRunningTime="2026-07-17 18:17:18.918616846 +0000 UTC m=+445.737388110" Jul 17 18:17:25.894544 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:25.894476 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86cc6d594-jwncj" podUID="b8776472-681b-45ea-a6ca-2d6893f080dc" containerName="console" containerID="cri-o://7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019" gracePeriod=15 Jul 17 18:17:26.148703 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.148638 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:17:26.294813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.294778 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.294813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.294822 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295092 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.294873 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295092 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.294917 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295092 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.294961 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295092 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295017 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9x9\" (UniqueName: \"kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295092 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295041 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config\") pod \"b8776472-681b-45ea-a6ca-2d6893f080dc\" (UID: \"b8776472-681b-45ea-a6ca-2d6893f080dc\") " Jul 17 18:17:26.295355 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295320 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config" (OuterVolumeSpecName: "console-config") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:26.295414 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295370 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:26.295452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295405 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:26.295484 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.295444 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:26.297174 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.297154 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:17:26.297572 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.297549 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:17:26.297634 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.297564 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9" (OuterVolumeSpecName: "kube-api-access-rd9x9") pod "b8776472-681b-45ea-a6ca-2d6893f080dc" (UID: "b8776472-681b-45ea-a6ca-2d6893f080dc"). InnerVolumeSpecName "kube-api-access-rd9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:26.396201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396150 2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-service-ca\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396195 2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-trusted-ca-bundle\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396206 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9x9\" (UniqueName: \"kubernetes.io/projected/b8776472-681b-45ea-a6ca-2d6893f080dc-kube-api-access-rd9x9\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396216 2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-oauth-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396201 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396224 2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-oauth-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396495 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396233 2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8776472-681b-45ea-a6ca-2d6893f080dc-console-serving-cert\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.396495 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.396241 2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8776472-681b-45ea-a6ca-2d6893f080dc-console-config\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:26.932124 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.932079 2587 generic.go:356] "Generic (PLEG): container finished" podID="b8776472-681b-45ea-a6ca-2d6893f080dc" containerID="7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019" exitCode=0 Jul 17 18:17:26.932553 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.932127 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc6d594-jwncj" Jul 17 18:17:26.932553 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.932171 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc6d594-jwncj" event={"ID":"b8776472-681b-45ea-a6ca-2d6893f080dc","Type":"ContainerDied","Data":"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019"} Jul 17 18:17:26.932553 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.932219 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc6d594-jwncj" event={"ID":"b8776472-681b-45ea-a6ca-2d6893f080dc","Type":"ContainerDied","Data":"79d614ab33d13015fd5a344a92c7fa11968e9fd5feea42da6cc8a4838ebd2199"} Jul 17 18:17:26.932553 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.932241 2587 scope.go:117] "RemoveContainer" containerID="7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019" Jul 17 18:17:26.941526 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.941512 2587 scope.go:117] "RemoveContainer" containerID="7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019" Jul 17 18:17:26.941781 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:17:26.941762 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019\": container with ID starting with 7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019 not found: ID does not exist" containerID="7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019" Jul 17 18:17:26.941840 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.941789 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019"} err="failed to get container status \"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019\": rpc error: code = NotFound desc = could not find container \"7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019\": container with ID starting with 7e6ab777f941a53e09ccb68e3ecf2baf91714e681978e15ecf9b772056b5b019 not found: ID does not exist" Jul 17 18:17:26.956087 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.956061 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:17:26.962407 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:26.962381 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86cc6d594-jwncj"] Jul 17 18:17:27.743177 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:27.741206 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8776472-681b-45ea-a6ca-2d6893f080dc" path="/var/lib/kubelet/pods/b8776472-681b-45ea-a6ca-2d6893f080dc/volumes" Jul 17 18:17:43.008411 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.008375 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:17:43.008932 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.008870 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8776472-681b-45ea-a6ca-2d6893f080dc" containerName="console" Jul 17 18:17:43.008932 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.008890 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8776472-681b-45ea-a6ca-2d6893f080dc" containerName="console" Jul 17 18:17:43.009097 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.009028 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8776472-681b-45ea-a6ca-2d6893f080dc" containerName="console" Jul 17 18:17:43.091604 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.091568 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:17:43.091748 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.091678 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.096320 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.096296 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 18:17:43.132487 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.132462 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:17:43.239560 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.239517 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-config-file\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.239743 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.239596 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrj5\" (UniqueName: \"kubernetes.io/projected/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-kube-api-access-gsrj5\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.340692 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.340616 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrj5\" (UniqueName: \"kubernetes.io/projected/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-kube-api-access-gsrj5\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.340913 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.340796 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-config-file\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.341408 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.341391 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-config-file\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.354694 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.354659 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrj5\" (UniqueName: \"kubernetes.io/projected/fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa-kube-api-access-gsrj5\") pod \"limitador-limitador-54d6d988d5-4xqw9\" (UID: \"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.401860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.401824 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:43.552627 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.552593 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:17:43.554519 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:17:43.554489 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfede4f0f_ce6d_4702_9e4c_d48aa6d7d7aa.slice/crio-6ed531e6aa520f4589bfe73467e683005117a29b7da66c7c3be250f77c69c5f3 WatchSource:0}: Error finding container 6ed531e6aa520f4589bfe73467e683005117a29b7da66c7c3be250f77c69c5f3: Status 404 returned error can't find the container with id 6ed531e6aa520f4589bfe73467e683005117a29b7da66c7c3be250f77c69c5f3 Jul 17 18:17:43.847669 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.847637 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:43.852224 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.852206 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:43.857860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.857832 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-88ddz\"" Jul 17 18:17:43.871759 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.871738 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:43.944971 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:43.944937 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrxq\" (UniqueName: \"kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq\") pod \"authorino-58cd955675-bbcjz\" (UID: \"1cf2f638-62e6-43f4-8b84-4cb054e4d348\") " pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:44.011385 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:44.011352 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" event={"ID":"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa","Type":"ContainerStarted","Data":"6ed531e6aa520f4589bfe73467e683005117a29b7da66c7c3be250f77c69c5f3"} Jul 17 18:17:44.045824 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:44.045795 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrxq\" (UniqueName: \"kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq\") pod \"authorino-58cd955675-bbcjz\" (UID: \"1cf2f638-62e6-43f4-8b84-4cb054e4d348\") " pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:44.071815 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:44.071785 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrxq\" (UniqueName: \"kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq\") pod \"authorino-58cd955675-bbcjz\" (UID: \"1cf2f638-62e6-43f4-8b84-4cb054e4d348\") " pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:44.160875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:44.160800 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:44.366532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:44.365367 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:44.370564 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:17:44.370531 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cf2f638_62e6_43f4_8b84_4cb054e4d348.slice/crio-7e5ea4cdd69d23c0d9611c637c5e96c87096a84fad04ff24f04d281e64197c5a WatchSource:0}: Error finding container 7e5ea4cdd69d23c0d9611c637c5e96c87096a84fad04ff24f04d281e64197c5a: Status 404 returned error can't find the container with id 7e5ea4cdd69d23c0d9611c637c5e96c87096a84fad04ff24f04d281e64197c5a Jul 17 18:17:45.018864 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:45.018831 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bbcjz" event={"ID":"1cf2f638-62e6-43f4-8b84-4cb054e4d348","Type":"ContainerStarted","Data":"7e5ea4cdd69d23c0d9611c637c5e96c87096a84fad04ff24f04d281e64197c5a"} Jul 17 18:17:47.037687 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:47.037644 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" event={"ID":"fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa","Type":"ContainerStarted","Data":"a90387d5778826e6669c24de1aa6e0ba833d32ea8ed021797a9f4cc04a29631a"} Jul 17 18:17:47.038116 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:47.037756 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:17:47.074069 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:47.074022 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" podStartSLOduration=2.350044171 podStartE2EDuration="5.0740063s" totalImagesPullingTime="2.723962129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:47.07391216 +0000 UTC m=+473.892683425" watchObservedRunningTime="2026-07-17 18:17:47.0740063 +0000 UTC m=+473.892777557" Jul 17 18:17:49.051144 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:49.051106 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bbcjz" event={"ID":"1cf2f638-62e6-43f4-8b84-4cb054e4d348","Type":"ContainerStarted","Data":"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9"} Jul 17 18:17:49.085750 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:49.085682 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-bbcjz" podStartSLOduration=1.97640388 podStartE2EDuration="6.085664618s" totalImagesPullingTime="4.109260738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:49.085027437 +0000 UTC m=+475.903798702" watchObservedRunningTime="2026-07-17 18:17:49.085664618 +0000 UTC m=+475.904435881" Jul 17 18:17:49.319681 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:49.319601 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:51.060619 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:51.060550 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-bbcjz" podUID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" containerName="authorino" containerID="cri-o://2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9" gracePeriod=30 Jul 17 18:17:51.312943 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:51.312883 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:51.408331 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:51.408299 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hrxq\" (UniqueName: \"kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq\") pod \"1cf2f638-62e6-43f4-8b84-4cb054e4d348\" (UID: \"1cf2f638-62e6-43f4-8b84-4cb054e4d348\") " Jul 17 18:17:51.410452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:51.410426 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq" (OuterVolumeSpecName: "kube-api-access-2hrxq") pod "1cf2f638-62e6-43f4-8b84-4cb054e4d348" (UID: "1cf2f638-62e6-43f4-8b84-4cb054e4d348"). InnerVolumeSpecName "kube-api-access-2hrxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:51.509249 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:51.509210 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrxq\" (UniqueName: \"kubernetes.io/projected/1cf2f638-62e6-43f4-8b84-4cb054e4d348-kube-api-access-2hrxq\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:17:52.066514 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.066476 2587 generic.go:356] "Generic (PLEG): container finished" podID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" containerID="2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9" exitCode=0 Jul 17 18:17:52.066514 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.066519 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bbcjz" Jul 17 18:17:52.066968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.066532 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bbcjz" event={"ID":"1cf2f638-62e6-43f4-8b84-4cb054e4d348","Type":"ContainerDied","Data":"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9"} Jul 17 18:17:52.066968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.066576 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bbcjz" event={"ID":"1cf2f638-62e6-43f4-8b84-4cb054e4d348","Type":"ContainerDied","Data":"7e5ea4cdd69d23c0d9611c637c5e96c87096a84fad04ff24f04d281e64197c5a"} Jul 17 18:17:52.066968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.066592 2587 scope.go:117] "RemoveContainer" containerID="2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9" Jul 17 18:17:52.074818 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.074795 2587 scope.go:117] "RemoveContainer" containerID="2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9" Jul 17 18:17:52.075113 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:17:52.075084 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9\": container with ID starting with 2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9 not found: ID does not exist" containerID="2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9" Jul 17 18:17:52.075202 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.075120 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9"} err="failed to get container status \"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9\": rpc error: code = NotFound desc = could not find container \"2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9\": container with ID starting with 2f075720f0cbc04b8335d47a02bb3fd91686e83d34f985f243b42fbb42575ea9 not found: ID does not exist" Jul 17 18:17:52.100902 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.100872 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:52.108310 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:52.108284 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-bbcjz"] Jul 17 18:17:53.740303 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:53.740265 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" path="/var/lib/kubelet/pods/1cf2f638-62e6-43f4-8b84-4cb054e4d348/volumes" Jul 17 18:17:58.045956 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:17:58.045918 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4xqw9" Jul 17 18:18:16.957438 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.957400 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:16.958106 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.957760 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" containerName="authorino" Jul 17 18:18:16.958106 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.957772 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" containerName="authorino" Jul 17 18:18:16.958106 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.957831 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf2f638-62e6-43f4-8b84-4cb054e4d348" containerName="authorino" Jul 17 18:18:16.960333 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.960314 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:16.962586 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.962564 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-88ddz\"" Jul 17 18:18:16.970007 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:16.969970 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:17.016599 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.016560 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2j8\" (UniqueName: \"kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8\") pod \"authorino-5d4b67c446-dzz8f\" (UID: \"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829\") " pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:17.117354 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.117308 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2j8\" (UniqueName: \"kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8\") pod \"authorino-5d4b67c446-dzz8f\" (UID: \"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829\") " pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:17.127566 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.127540 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2j8\" (UniqueName: \"kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8\") pod \"authorino-5d4b67c446-dzz8f\" (UID: \"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829\") " pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:17.205681 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.205644 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:17.205907 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.205895 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:17.233884 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.233853 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7955bfcdbb-bwksw"] Jul 17 18:18:17.237791 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.237768 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:17.245508 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.245481 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7955bfcdbb-bwksw"] Jul 17 18:18:17.324806 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.324762 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzswx\" (UniqueName: \"kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx\") pod \"authorino-7955bfcdbb-bwksw\" (UID: \"3f312f41-dce2-4a20-a643-ccc231c0802c\") " pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:17.327383 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.327357 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7955bfcdbb-bwksw"] Jul 17 18:18:17.327580 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:18:17.327561 2587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tzswx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7955bfcdbb-bwksw" podUID="3f312f41-dce2-4a20-a643-ccc231c0802c" Jul 17 18:18:17.342467 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.342446 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:17.344901 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:18:17.344869 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e36a7a9_8d6f_4ccc_ac07_211ebd8f7829.slice/crio-4982a8cd0b806f6574a165e4cb5cb4f29839ff470acbb809dfaddee077afc4c9 WatchSource:0}: Error finding container 4982a8cd0b806f6574a165e4cb5cb4f29839ff470acbb809dfaddee077afc4c9: Status 404 returned error can't find the container with id 4982a8cd0b806f6574a165e4cb5cb4f29839ff470acbb809dfaddee077afc4c9 Jul 17 18:18:17.425495 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.425459 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzswx\" (UniqueName: \"kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx\") pod \"authorino-7955bfcdbb-bwksw\" (UID: \"3f312f41-dce2-4a20-a643-ccc231c0802c\") " pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:17.434189 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:17.434165 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzswx\" (UniqueName: \"kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx\") pod \"authorino-7955bfcdbb-bwksw\" (UID: \"3f312f41-dce2-4a20-a643-ccc231c0802c\") " pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:18.182490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.182455 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" event={"ID":"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829","Type":"ContainerStarted","Data":"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5"} Jul 17 18:18:18.182490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.182488 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" podUID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" containerName="authorino" containerID="cri-o://db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5" gracePeriod=30 Jul 17 18:18:18.182976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.182501 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" event={"ID":"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829","Type":"ContainerStarted","Data":"4982a8cd0b806f6574a165e4cb5cb4f29839ff470acbb809dfaddee077afc4c9"} Jul 17 18:18:18.182976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.182494 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:18.191123 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.191102 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:18.198056 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.198012 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" podStartSLOduration=1.83445078 podStartE2EDuration="2.197972854s" totalImagesPullingTime="363.522074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:18:18.196143448 +0000 UTC m=+505.014914711" watchObservedRunningTime="2026-07-17 18:18:18.197972854 +0000 UTC m=+505.016744118" Jul 17 18:18:18.232183 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.232144 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzswx\" (UniqueName: \"kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx\") pod \"3f312f41-dce2-4a20-a643-ccc231c0802c\" (UID: \"3f312f41-dce2-4a20-a643-ccc231c0802c\") " Jul 17 18:18:18.234240 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.234214 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx" (OuterVolumeSpecName: "kube-api-access-tzswx") pod "3f312f41-dce2-4a20-a643-ccc231c0802c" (UID: "3f312f41-dce2-4a20-a643-ccc231c0802c"). InnerVolumeSpecName "kube-api-access-tzswx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:18:18.333935 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.333900 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzswx\" (UniqueName: \"kubernetes.io/projected/3f312f41-dce2-4a20-a643-ccc231c0802c-kube-api-access-tzswx\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:18:18.432581 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.432447 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:18.534828 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.534791 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv2j8\" (UniqueName: \"kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8\") pod \"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829\" (UID: \"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829\") " Jul 17 18:18:18.536860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.536838 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8" (OuterVolumeSpecName: "kube-api-access-xv2j8") pod "2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" (UID: "2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829"). InnerVolumeSpecName "kube-api-access-xv2j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:18:18.635599 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:18.635569 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv2j8\" (UniqueName: \"kubernetes.io/projected/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829-kube-api-access-xv2j8\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:18:19.188562 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188529 2587 generic.go:356] "Generic (PLEG): container finished" podID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" containerID="db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5" exitCode=0 Jul 17 18:18:19.189061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188577 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" Jul 17 18:18:19.189061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188621 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" event={"ID":"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829","Type":"ContainerDied","Data":"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5"} Jul 17 18:18:19.189061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188658 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-dzz8f" event={"ID":"2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829","Type":"ContainerDied","Data":"4982a8cd0b806f6574a165e4cb5cb4f29839ff470acbb809dfaddee077afc4c9"} Jul 17 18:18:19.189061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188675 2587 scope.go:117] "RemoveContainer" containerID="db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5" Jul 17 18:18:19.189061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.188816 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7955bfcdbb-bwksw" Jul 17 18:18:19.198440 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.198423 2587 scope.go:117] "RemoveContainer" containerID="db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5" Jul 17 18:18:19.198709 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:18:19.198691 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5\": container with ID starting with db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5 not found: ID does not exist" containerID="db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5" Jul 17 18:18:19.198764 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.198718 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5"} err="failed to get container status \"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5\": rpc error: code = NotFound desc = could not find container \"db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5\": container with ID starting with db23dd59ea9c87698d82a31f5592fa5ce0d0b913b3e547f73631aa2c07d516e5 not found: ID does not exist" Jul 17 18:18:19.219994 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.219950 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:19.228454 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.228432 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dzz8f"] Jul 17 18:18:19.257240 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.257210 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7955bfcdbb-bwksw"] Jul 17 18:18:19.264203 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.264174 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7955bfcdbb-bwksw"] Jul 17 18:18:19.739306 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.739271 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" path="/var/lib/kubelet/pods/2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829/volumes" Jul 17 18:18:19.739601 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:19.739588 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f312f41-dce2-4a20-a643-ccc231c0802c" path="/var/lib/kubelet/pods/3f312f41-dce2-4a20-a643-ccc231c0802c/volumes" Jul 17 18:18:59.944244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:18:59.944204 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:19:17.886935 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:17.886900 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:19:22.783053 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:22.783015 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:19:30.869780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.869749 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26"] Jul 17 18:19:30.870226 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.870133 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" containerName="authorino" Jul 17 18:19:30.870226 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.870145 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" containerName="authorino" Jul 17 18:19:30.870226 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.870198 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e36a7a9-8d6f-4ccc-ac07-211ebd8f7829" containerName="authorino" Jul 17 18:19:30.873195 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.873179 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.875995 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.875967 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 18:19:30.876085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.875967 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 18:19:30.876085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.875973 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rp4s8\"" Jul 17 18:19:30.876168 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.876125 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 18:19:30.880098 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880077 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.880190 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880107 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.880190 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880144 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.880277 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880239 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfb24\" (UniqueName: \"kubernetes.io/projected/e85d146a-83ff-49a3-9c73-3bda0f4f82de-kube-api-access-pfb24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.880331 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880314 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.880382 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.880367 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.888056 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.888036 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26"] Jul 17 18:19:30.980940 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.980904 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfb24\" (UniqueName: \"kubernetes.io/projected/e85d146a-83ff-49a3-9c73-3bda0f4f82de-kube-api-access-pfb24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.980965 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981028 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981128 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981064 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981128 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981086 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981128 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981123 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981491 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981462 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981491 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981482 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.981648 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.981530 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.983136 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.983118 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e85d146a-83ff-49a3-9c73-3bda0f4f82de-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.983508 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.983489 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e85d146a-83ff-49a3-9c73-3bda0f4f82de-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:30.991426 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:30.991408 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfb24\" (UniqueName: \"kubernetes.io/projected/e85d146a-83ff-49a3-9c73-3bda0f4f82de-kube-api-access-pfb24\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dj26\" (UID: \"e85d146a-83ff-49a3-9c73-3bda0f4f82de\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:31.182621 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:31.182560 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:31.308519 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:31.308495 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26"] Jul 17 18:19:31.310576 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:19:31.310551 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85d146a_83ff_49a3_9c73_3bda0f4f82de.slice/crio-f93404f34afad77dc5f6a7d06f3ae4883e410555ae673b507de1b8db100341da WatchSource:0}: Error finding container f93404f34afad77dc5f6a7d06f3ae4883e410555ae673b507de1b8db100341da: Status 404 returned error can't find the container with id f93404f34afad77dc5f6a7d06f3ae4883e410555ae673b507de1b8db100341da Jul 17 18:19:31.519109 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:31.519067 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" event={"ID":"e85d146a-83ff-49a3-9c73-3bda0f4f82de","Type":"ContainerStarted","Data":"f93404f34afad77dc5f6a7d06f3ae4883e410555ae673b507de1b8db100341da"} Jul 17 18:19:31.978443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:31.978369 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:19:32.524948 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:32.524915 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" event={"ID":"e85d146a-83ff-49a3-9c73-3bda0f4f82de","Type":"ContainerStarted","Data":"9c152783239f44f5b5cc225a16e85253370a93e81d5c1a0081952f08a0cc9d14"} Jul 17 18:19:32.541550 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:32.541500 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" podStartSLOduration=1.5581285280000001 podStartE2EDuration="2.541484095s" totalImagesPullingTime="983.355567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:32.539900654 +0000 UTC m=+579.358671941" watchObservedRunningTime="2026-07-17 18:19:32.541484095 +0000 UTC m=+579.360255388" Jul 17 18:19:41.183176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:41.183133 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:41.183573 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:41.183203 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:41.195729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:41.195704 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:41.579206 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:41.579176 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dj26" Jul 17 18:19:53.996657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:53.996549 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:19:53.996657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:19:53.996567 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:20:05.469814 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:05.469781 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:20:20.055164 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.055129 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw"] Jul 17 18:20:20.059302 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.059272 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.061374 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.061346 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 18:20:20.068374 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.068346 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw"] Jul 17 18:20:20.135530 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135488 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.135718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135545 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.135718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135602 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlmp\" (UniqueName: \"kubernetes.io/projected/75e59e2d-85f9-41f9-b171-9ea6616b65c0-kube-api-access-zwlmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.135718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135634 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.135718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135670 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.135855 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.135719 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236582 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236544 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236582 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236586 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236623 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236668 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236690 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.236787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.236746 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlmp\" (UniqueName: \"kubernetes.io/projected/75e59e2d-85f9-41f9-b171-9ea6616b65c0-kube-api-access-zwlmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.237123 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.237099 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.237209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.237146 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.237209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.237157 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.238871 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.238850 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e59e2d-85f9-41f9-b171-9ea6616b65c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.239199 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.239179 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e59e2d-85f9-41f9-b171-9ea6616b65c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.244472 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.244451 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlmp\" (UniqueName: \"kubernetes.io/projected/75e59e2d-85f9-41f9-b171-9ea6616b65c0-kube-api-access-zwlmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g79jw\" (UID: \"75e59e2d-85f9-41f9-b171-9ea6616b65c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.370200 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.370102 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:20.511749 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.511724 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw"] Jul 17 18:20:20.514067 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:20:20.514036 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e59e2d_85f9_41f9_b171_9ea6616b65c0.slice/crio-329f88976ef3375e59a659c24a065f82b2dcd098c0847693ac380e5ce9a9061a WatchSource:0}: Error finding container 329f88976ef3375e59a659c24a065f82b2dcd098c0847693ac380e5ce9a9061a: Status 404 returned error can't find the container with id 329f88976ef3375e59a659c24a065f82b2dcd098c0847693ac380e5ce9a9061a Jul 17 18:20:20.515359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.515339 2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:20:20.749049 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:20.749011 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" event={"ID":"75e59e2d-85f9-41f9-b171-9ea6616b65c0","Type":"ContainerStarted","Data":"329f88976ef3375e59a659c24a065f82b2dcd098c0847693ac380e5ce9a9061a"} Jul 17 18:20:21.754536 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:21.754500 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" event={"ID":"75e59e2d-85f9-41f9-b171-9ea6616b65c0","Type":"ContainerStarted","Data":"a6a9461c8ee20c75d04b07c9d22c1a23fc404006d96c991b5f1d4c5363b83f1c"} Jul 17 18:20:21.786818 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:21.786765 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" podStartSLOduration=1.60541464 podStartE2EDuration="1.786745131s" totalImagesPullingTime="181.330491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:20:21.785035932 +0000 UTC m=+628.603807196" watchObservedRunningTime="2026-07-17 18:20:21.786745131 +0000 UTC m=+628.605516394" Jul 17 18:20:22.172051 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:22.172015 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:20:26.775280 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.775241 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd"] Jul 17 18:20:26.779442 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.779418 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.781551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.781530 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 18:20:26.789262 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.789239 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd"] Jul 17 18:20:26.905333 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905298 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.905505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905358 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.905505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905382 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.905505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905436 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmbx\" (UniqueName: \"kubernetes.io/projected/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-kube-api-access-kcmbx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.905505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905493 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:26.905647 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:26.905510 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.005972 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.005932 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.005976 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006064 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006128 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006152 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006373 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006188 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmbx\" (UniqueName: \"kubernetes.io/projected/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-kube-api-access-kcmbx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006451 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006425 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006443 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.006551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.006519 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.008376 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.008358 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.008640 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.008619 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.014114 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.014085 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmbx\" (UniqueName: \"kubernetes.io/projected/a4d56e3b-71ce-4a02-8134-b8d08c6e6856-kube-api-access-kcmbx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd\" (UID: \"a4d56e3b-71ce-4a02-8134-b8d08c6e6856\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.091228 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.091146 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:27.221725 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.221701 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd"] Jul 17 18:20:27.224057 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:20:27.224023 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d56e3b_71ce_4a02_8134_b8d08c6e6856.slice/crio-aca1ffd78f46a52a0fd16c07eea3bed5d844a8012e2d40da56b65043efc0d1af WatchSource:0}: Error finding container aca1ffd78f46a52a0fd16c07eea3bed5d844a8012e2d40da56b65043efc0d1af: Status 404 returned error can't find the container with id aca1ffd78f46a52a0fd16c07eea3bed5d844a8012e2d40da56b65043efc0d1af Jul 17 18:20:27.783077 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.783037 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" event={"ID":"a4d56e3b-71ce-4a02-8134-b8d08c6e6856","Type":"ContainerStarted","Data":"8326a97f9e1ce6e26325c32e1c77f169e33b40a38cf1322d0e0ca3aba058c0d6"} Jul 17 18:20:27.783473 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.783089 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" event={"ID":"a4d56e3b-71ce-4a02-8134-b8d08c6e6856","Type":"ContainerStarted","Data":"aca1ffd78f46a52a0fd16c07eea3bed5d844a8012e2d40da56b65043efc0d1af"} Jul 17 18:20:27.799669 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:27.799609 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" podStartSLOduration=1.643649737 podStartE2EDuration="1.799586054s" totalImagesPullingTime="155.936317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:20:27.798615485 +0000 UTC m=+634.617386748" watchObservedRunningTime="2026-07-17 18:20:27.799586054 +0000 UTC m=+634.618357317" Jul 17 18:20:30.370331 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:30.370298 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:30.370709 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:30.370571 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:30.382239 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:30.382216 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:30.806018 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:30.805972 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g79jw" Jul 17 18:20:36.483193 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:36.483159 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:20:37.091278 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:37.091245 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:37.091537 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:37.091516 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:37.103444 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:37.103419 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:20:37.834600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:20:37.834574 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd" Jul 17 18:21:48.978633 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:21:48.978557 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:22:00.623365 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:22:00.623325 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:22:11.218973 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:22:11.218936 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:23:53.181473 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:23:53.181443 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:24:05.574934 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:05.574899 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:24:41.170532 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:41.170494 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:24:50.475787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:50.475749 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:24:54.026167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:54.026137 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:24:54.028483 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:54.026793 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:24:59.489073 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:24:59.489037 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:25:14.984497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:25:14.984458 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:25:25.377275 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:25:25.377234 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:25:34.783131 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:25:34.783094 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:25:46.886270 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:25:46.886227 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:25:56.978919 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:25:56.978881 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:26:22.282045 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:26:22.282002 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:26:33.881613 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:26:33.881581 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:26:58.888887 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:26:58.888847 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:27:09.675609 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:27:09.675572 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:27:27.279710 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:27:27.279669 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:27:41.970419 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:27:41.970385 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:27:56.177851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:27:56.177765 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:28:05.683246 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:28:05.683206 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:29:54.057880 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:29:54.057787 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:29:54.061786 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:29:54.059777 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:29:54.282388 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:29:54.282356 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:04.587565 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:04.587518 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:14.906071 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:14.906035 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:28.617696 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:28.617661 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:38.521745 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:38.521712 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:48.297301 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:48.297224 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:30:56.686267 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:30:56.686231 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:31:06.583712 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:31:06.583677 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:31:17.796180 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:31:17.796147 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:31:27.984268 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:31:27.984233 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:31:40.384341 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:31:40.384303 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:32:04.590892 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:32:04.590850 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:32:14.278219 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:32:14.278183 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:32:24.187415 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:32:24.187379 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:32:37.970863 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:32:37.970827 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:33:15.697459 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:33:15.697422 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:33:24.986615 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:33:24.986582 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:33:35.188899 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:33:35.188866 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:33:45.086174 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:33:45.086135 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:33:55.488884 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:33:55.488848 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:34:05.979653 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:05.979615 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:34:17.983603 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:17.983564 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:34:33.283976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:33.283930 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:34:42.885850 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:42.885808 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:34:54.085972 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:54.085864 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:34:54.090110 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:54.090095 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:34:54.275520 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:34:54.275480 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:35:11.929155 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:35:11.929120 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:35:26.789417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:35:26.789381 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:35:39.093125 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:35:39.093088 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:35:53.582152 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:35:53.582118 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:36:02.883016 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:36:02.881813 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:36:12.481828 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:36:12.481737 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:36:24.377423 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:36:24.377389 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:37:31.784670 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:37:31.784633 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:37:54.581270 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:37:54.581175 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:38:53.192209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:38:53.192170 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:39:06.287639 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:39:06.287603 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:39:54.119148 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:39:54.119040 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:39:54.125678 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:39:54.121717 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:40:12.176505 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:40:12.176463 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:40:34.182610 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:40:34.182574 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:41:44.089456 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:41:44.089416 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:43:21.979890 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:43:21.979853 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:43:31.081255 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:43:31.081219 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:43:45.277868 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:43:45.277835 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:44:03.078141 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:44:03.078107 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:44:43.579065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:44:43.579030 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:44:54.146826 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:44:54.146717 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:44:54.151713 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:44:54.149881 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:45:02.481233 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.481199 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:45:02.484730 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.484709 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.487713 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.487688 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5a93a056\"/\"e2e-lifecycle-model-5a93a056-kserve-self-signed-certs\"" Jul 17 18:45:02.487826 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.487712 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5a93a056\"/\"default-dockercfg-7hjw4\"" Jul 17 18:45:02.487826 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.487769 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5a93a056\"/\"kube-root-ca.crt\"" Jul 17 18:45:02.487923 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.487896 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5a93a056\"/\"openshift-service-ca.crt\"" Jul 17 18:45:02.572658 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.572626 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:45:02.603871 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.603832 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.604085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.603896 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.604085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.603933 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.604085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.604005 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.604085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.604036 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.604085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.604071 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzs25\" (UniqueName: \"kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.704828 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.704789 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.704870 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.704907 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.704951 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzs25\" (UniqueName: \"kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705208 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.705036 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705208 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.705094 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705446 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.705410 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705562 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.705448 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.705562 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.705494 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.707342 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.707316 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.707724 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.707708 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.713362 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.713343 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzs25\" (UniqueName: \"kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25\") pod \"e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.794014 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.793963 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:02.928271 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:02.928247 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:45:02.930351 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:45:02.930325 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93462e0b_bd97_423a_bab2_dce4ff1b08ab.slice/crio-7a992f54e5455e953d7cf6457ef23d10ce25440faa474edef858c6c6218b7863 WatchSource:0}: Error finding container 7a992f54e5455e953d7cf6457ef23d10ce25440faa474edef858c6c6218b7863: Status 404 returned error can't find the container with id 7a992f54e5455e953d7cf6457ef23d10ce25440faa474edef858c6c6218b7863 Jul 17 18:45:03.630108 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:03.630074 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" event={"ID":"93462e0b-bd97-423a-bab2-dce4ff1b08ab","Type":"ContainerStarted","Data":"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673"} Jul 17 18:45:03.630108 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:03.630113 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" event={"ID":"93462e0b-bd97-423a-bab2-dce4ff1b08ab","Type":"ContainerStarted","Data":"7a992f54e5455e953d7cf6457ef23d10ce25440faa474edef858c6c6218b7863"} Jul 17 18:45:03.648073 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:03.648031 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" podStartSLOduration=1.648015471 podStartE2EDuration="1.648015471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:03.646853009 +0000 UTC m=+2110.465624275" watchObservedRunningTime="2026-07-17 18:45:03.648015471 +0000 UTC m=+2110.466786712" Jul 17 18:45:05.619866 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:05.619831 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:45:12.794132 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:12.794097 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:12.794132 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:12.794142 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:12.806553 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:12.806530 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:13.685366 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:13.685339 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:45:22.896375 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:22.896342 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:45:40.474845 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:40.474809 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:45:46.419183 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:45:46.419108 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:46:01.438574 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.438528 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" podUID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" containerName="main" containerID="cri-o://1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673" gracePeriod=45 Jul 17 18:46:01.683050 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.683023 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:46:01.740945 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.740883 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.740945 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.740912 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzs25\" (UniqueName: \"kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.740945 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.740940 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.741213 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.740972 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.741213 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741034 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.741213 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741094 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs\") pod \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\" (UID: \"93462e0b-bd97-423a-bab2-dce4ff1b08ab\") " Jul 17 18:46:01.741417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741263 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:46:01.741417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741289 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home" (OuterVolumeSpecName: "home") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:46:01.741417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741321 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:46:01.741417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741388 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.741417 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.741408 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.743349 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.743322 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm" (OuterVolumeSpecName: "dshm") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:46:01.743516 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.743495 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:46:01.743516 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.743507 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25" (OuterVolumeSpecName: "kube-api-access-lzs25") pod "93462e0b-bd97-423a-bab2-dce4ff1b08ab" (UID: "93462e0b-bd97-423a-bab2-dce4ff1b08ab"). InnerVolumeSpecName "kube-api-access-lzs25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:46:01.842361 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.842335 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.842361 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.842359 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzs25\" (UniqueName: \"kubernetes.io/projected/93462e0b-bd97-423a-bab2-dce4ff1b08ab-kube-api-access-lzs25\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.842519 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.842370 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93462e0b-bd97-423a-bab2-dce4ff1b08ab-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.842519 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.842378 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93462e0b-bd97-423a-bab2-dce4ff1b08ab-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:46:01.898245 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.898218 2587 generic.go:356] "Generic (PLEG): container finished" podID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" containerID="1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673" exitCode=0 Jul 17 18:46:01.898350 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.898260 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" Jul 17 18:46:01.898350 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.898299 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" event={"ID":"93462e0b-bd97-423a-bab2-dce4ff1b08ab","Type":"ContainerDied","Data":"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673"} Jul 17 18:46:01.898350 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.898344 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d" event={"ID":"93462e0b-bd97-423a-bab2-dce4ff1b08ab","Type":"ContainerDied","Data":"7a992f54e5455e953d7cf6457ef23d10ce25440faa474edef858c6c6218b7863"} Jul 17 18:46:01.898463 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.898366 2587 scope.go:117] "RemoveContainer" containerID="1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673" Jul 17 18:46:01.907391 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.907374 2587 scope.go:117] "RemoveContainer" containerID="1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673" Jul 17 18:46:01.907636 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:46:01.907615 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673\": container with ID starting with 1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673 not found: ID does not exist" containerID="1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673" Jul 17 18:46:01.907718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.907642 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673"} err="failed to get container status \"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673\": rpc error: code = NotFound desc = could not find container \"1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673\": container with ID starting with 1acfa02c3a92228b4d2f66d781676dab21b15ee2b6542fe0efadeadbd3568673 not found: ID does not exist" Jul 17 18:46:01.942780 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.942753 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:46:01.946358 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:01.946338 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5a93a056/e2e-lifecycle-model-5a93a056-kserve-58f5fd7-zj82d"] Jul 17 18:46:03.739767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:03.739738 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" path="/var/lib/kubelet/pods/93462e0b-bd97-423a-bab2-dce4ff1b08ab/volumes" Jul 17 18:46:17.623368 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:17.623327 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:46:31.285326 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:46:31.285291 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:47:20.082710 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:20.082674 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:47:31.473581 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:31.473543 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:47:34.195180 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.195146 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:47:34.195554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.195528 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" containerName="main" Jul 17 18:47:34.195554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.195538 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" containerName="main" Jul 17 18:47:34.195625 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.195597 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="93462e0b-bd97-423a-bab2-dce4ff1b08ab" containerName="main" Jul 17 18:47:34.198718 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.198697 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.200902 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.200879 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-764e6c\"/\"default-dockercfg-lmr24\"" Jul 17 18:47:34.201026 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.200915 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-764e6c\"/\"test-model-764e6c-kserve-self-signed-certs\"" Jul 17 18:47:34.201503 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.201487 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-764e6c\"/\"kube-root-ca.crt\"" Jul 17 18:47:34.201557 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.201509 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-764e6c\"/\"openshift-service-ca.crt\"" Jul 17 18:47:34.209596 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.209559 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:47:34.362811 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.362769 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.363029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.362822 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.363029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.362889 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.363029 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.362958 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglvt\" (UniqueName: \"kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.363171 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.363082 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.363171 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.363108 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464028 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464082 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464120 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464404 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464148 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglvt\" (UniqueName: \"kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464404 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464237 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464404 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464267 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464565 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464510 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464565 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464557 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.464691 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.464671 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.466367 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.466349 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.466537 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.466518 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.471571 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.471551 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglvt\" (UniqueName: \"kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt\") pod \"test-model-764e6c-kserve-7bcbc88c5f-hslvd\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.508345 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.508316 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:34.641469 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:34.641444 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:47:34.643741 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:47:34.643718 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod088f4624_e805_498f_b26f_ecac02d92f4b.slice/crio-db32b0ce9e1c7838376de6a8e0d464696fa9b2fb6c5a94361e962bf741cded53 WatchSource:0}: Error finding container db32b0ce9e1c7838376de6a8e0d464696fa9b2fb6c5a94361e962bf741cded53: Status 404 returned error can't find the container with id db32b0ce9e1c7838376de6a8e0d464696fa9b2fb6c5a94361e962bf741cded53 Jul 17 18:47:35.339879 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:35.339836 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" event={"ID":"088f4624-e805-498f-b26f-ecac02d92f4b","Type":"ContainerStarted","Data":"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57"} Jul 17 18:47:35.339879 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:35.339889 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" event={"ID":"088f4624-e805-498f-b26f-ecac02d92f4b","Type":"ContainerStarted","Data":"db32b0ce9e1c7838376de6a8e0d464696fa9b2fb6c5a94361e962bf741cded53"} Jul 17 18:47:35.358051 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:35.357970 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" podStartSLOduration=1.357952049 podStartE2EDuration="1.357952049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:35.356287679 +0000 UTC m=+2262.175058943" watchObservedRunningTime="2026-07-17 18:47:35.357952049 +0000 UTC m=+2262.176723312" Jul 17 18:47:44.191925 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:44.191892 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:47:44.509292 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:44.509260 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:44.509447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:44.509316 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:44.521796 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:44.521769 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:45.395795 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:45.395765 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:47:53.280821 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.280784 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:47:53.284492 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.284476 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.287228 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.287206 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8f4385\"/\"openshift-service-ca.crt\"" Jul 17 18:47:53.287228 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.287217 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8f4385\"/\"default-dockercfg-zwmk2\"" Jul 17 18:47:53.287412 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.287244 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8f4385\"/\"kube-root-ca.crt\"" Jul 17 18:47:53.287412 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.287223 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8f4385\"/\"test-model-8f4385-kserve-self-signed-certs\"" Jul 17 18:47:53.295077 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.295056 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:47:53.319314 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319276 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.319492 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319362 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9rv\" (UniqueName: \"kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.319492 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319407 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.319492 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319445 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.319492 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319469 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.319651 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.319512 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.419997 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9rv\" (UniqueName: \"kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420050 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420085 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420102 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420120 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420223 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420157 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420592 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420569 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420572 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.420655 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.420619 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.422787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.422761 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.422787 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.422776 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.428586 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.428559 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9rv\" (UniqueName: \"kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv\") pod \"test-model-8f4385-kserve-546b4866b9-8wjrg\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.597287 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.597210 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8f4385\"/\"default-dockercfg-zwmk2\"" Jul 17 18:47:53.604627 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.604587 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:47:53.745722 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:53.745698 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:47:53.748113 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:47:53.748083 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fb3bf02_62ad_41af_8158_8f283d76e5c5.slice/crio-2310cb337e506da24e8af02188c7252fd4f678bc2140c2edb1d9b710bdc3de74 WatchSource:0}: Error finding container 2310cb337e506da24e8af02188c7252fd4f678bc2140c2edb1d9b710bdc3de74: Status 404 returned error can't find the container with id 2310cb337e506da24e8af02188c7252fd4f678bc2140c2edb1d9b710bdc3de74 Jul 17 18:47:54.433104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:54.433071 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" event={"ID":"4fb3bf02-62ad-41af-8158-8f283d76e5c5","Type":"ContainerStarted","Data":"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9"} Jul 17 18:47:54.433479 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:54.433117 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" event={"ID":"4fb3bf02-62ad-41af-8158-8f283d76e5c5","Type":"ContainerStarted","Data":"2310cb337e506da24e8af02188c7252fd4f678bc2140c2edb1d9b710bdc3de74"} Jul 17 18:47:54.463330 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:54.463272 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" podStartSLOduration=1.463253904 podStartE2EDuration="1.463253904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:54.460780263 +0000 UTC m=+2281.279551524" watchObservedRunningTime="2026-07-17 18:47:54.463253904 +0000 UTC m=+2281.282025168" Jul 17 18:47:56.987917 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:47:56.987879 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:48:03.604838 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:03.604794 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:03.605372 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:03.604860 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:03.617242 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:03.617215 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:04.492937 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:04.492902 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:13.009614 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:13.009580 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:48:13.886313 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:13.886279 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:48:16.087761 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:16.087713 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:48:28.029590 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.029549 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" podUID="088f4624-e805-498f-b26f-ecac02d92f4b" containerName="main" containerID="cri-o://4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57" gracePeriod=45 Jul 17 18:48:28.277707 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.277675 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:48:28.344808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344716 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zglvt\" (UniqueName: \"kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.344808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344780 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.345059 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344819 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.345059 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344859 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.345059 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344904 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.345059 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.344925 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache\") pod \"088f4624-e805-498f-b26f-ecac02d92f4b\" (UID: \"088f4624-e805-498f-b26f-ecac02d92f4b\") " Jul 17 18:48:28.345260 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.345086 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home" (OuterVolumeSpecName: "home") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:28.345260 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.345174 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:28.345359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.345269 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:28.345359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.345291 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.345359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.345306 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.347162 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.347132 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:48:28.347162 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.347152 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm" (OuterVolumeSpecName: "dshm") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:28.347294 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.347202 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt" (OuterVolumeSpecName: "kube-api-access-zglvt") pod "088f4624-e805-498f-b26f-ecac02d92f4b" (UID: "088f4624-e805-498f-b26f-ecac02d92f4b"). InnerVolumeSpecName "kube-api-access-zglvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:48:28.446678 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.446644 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088f4624-e805-498f-b26f-ecac02d92f4b-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.446678 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.446672 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.446678 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.446681 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088f4624-e805-498f-b26f-ecac02d92f4b-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.446904 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.446690 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zglvt\" (UniqueName: \"kubernetes.io/projected/088f4624-e805-498f-b26f-ecac02d92f4b-kube-api-access-zglvt\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:28.598096 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.598013 2587 generic.go:356] "Generic (PLEG): container finished" podID="088f4624-e805-498f-b26f-ecac02d92f4b" containerID="4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57" exitCode=0 Jul 17 18:48:28.598096 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.598050 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" Jul 17 18:48:28.598096 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.598059 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" event={"ID":"088f4624-e805-498f-b26f-ecac02d92f4b","Type":"ContainerDied","Data":"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57"} Jul 17 18:48:28.598096 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.598097 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd" event={"ID":"088f4624-e805-498f-b26f-ecac02d92f4b","Type":"ContainerDied","Data":"db32b0ce9e1c7838376de6a8e0d464696fa9b2fb6c5a94361e962bf741cded53"} Jul 17 18:48:28.598346 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.598117 2587 scope.go:117] "RemoveContainer" containerID="4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57" Jul 17 18:48:28.608578 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.608557 2587 scope.go:117] "RemoveContainer" containerID="4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57" Jul 17 18:48:28.608830 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:48:28.608813 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57\": container with ID starting with 4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57 not found: ID does not exist" containerID="4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57" Jul 17 18:48:28.608900 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.608840 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57"} err="failed to get container status \"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57\": rpc error: code = NotFound desc = could not find container \"4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57\": container with ID starting with 4c0a4edb1507ea9fa083a1dbcd925afc455203153b0c5a5f6bec8415e9385c57 not found: ID does not exist" Jul 17 18:48:28.621565 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.621526 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:48:28.628089 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.628060 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-764e6c/test-model-764e6c-kserve-7bcbc88c5f-hslvd"] Jul 17 18:48:28.906362 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:28.906272 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" podUID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" containerName="main" containerID="cri-o://f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9" gracePeriod=45 Jul 17 18:48:29.156942 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.156884 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:29.253085 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253060 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff9rv\" (UniqueName: \"kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253243 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253098 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253243 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253131 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253319 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253289 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253366 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253352 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253409 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253364 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:29.253409 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253390 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir\") pod \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\" (UID: \"4fb3bf02-62ad-41af-8158-8f283d76e5c5\") " Jul 17 18:48:29.253620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253586 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home" (OuterVolumeSpecName: "home") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:29.253713 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253614 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:29.253772 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253723 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.253772 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253740 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.253772 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.253749 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.255276 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.255254 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm" (OuterVolumeSpecName: "dshm") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:48:29.255374 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.255342 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:48:29.255420 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.255404 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv" (OuterVolumeSpecName: "kube-api-access-ff9rv") pod "4fb3bf02-62ad-41af-8158-8f283d76e5c5" (UID: "4fb3bf02-62ad-41af-8158-8f283d76e5c5"). InnerVolumeSpecName "kube-api-access-ff9rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:48:29.354809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.354777 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb3bf02-62ad-41af-8158-8f283d76e5c5-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.354809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.354805 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff9rv\" (UniqueName: \"kubernetes.io/projected/4fb3bf02-62ad-41af-8158-8f283d76e5c5-kube-api-access-ff9rv\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.354809 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.354816 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb3bf02-62ad-41af-8158-8f283d76e5c5-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:48:29.604971 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.604939 2587 generic.go:356] "Generic (PLEG): container finished" podID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" containerID="f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9" exitCode=0 Jul 17 18:48:29.605154 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.605002 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" Jul 17 18:48:29.605154 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.605039 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" event={"ID":"4fb3bf02-62ad-41af-8158-8f283d76e5c5","Type":"ContainerDied","Data":"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9"} Jul 17 18:48:29.605154 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.605082 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg" event={"ID":"4fb3bf02-62ad-41af-8158-8f283d76e5c5","Type":"ContainerDied","Data":"2310cb337e506da24e8af02188c7252fd4f678bc2140c2edb1d9b710bdc3de74"} Jul 17 18:48:29.605154 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.605101 2587 scope.go:117] "RemoveContainer" containerID="f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9" Jul 17 18:48:29.614281 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.614263 2587 scope.go:117] "RemoveContainer" containerID="f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9" Jul 17 18:48:29.614515 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:48:29.614497 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9\": container with ID starting with f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9 not found: ID does not exist" containerID="f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9" Jul 17 18:48:29.614556 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.614522 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9"} err="failed to get container status \"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9\": rpc error: code = NotFound desc = could not find container \"f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9\": container with ID starting with f25a9b8e108d65874d4a5ea4243bd84befdd866a6e9f257f920e8ab8e45cada9 not found: ID does not exist" Jul 17 18:48:29.626797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.626777 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:48:29.631596 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.631575 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8f4385/test-model-8f4385-kserve-546b4866b9-8wjrg"] Jul 17 18:48:29.740918 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.740879 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088f4624-e805-498f-b26f-ecac02d92f4b" path="/var/lib/kubelet/pods/088f4624-e805-498f-b26f-ecac02d92f4b/volumes" Jul 17 18:48:29.741358 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:29.741343 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" path="/var/lib/kubelet/pods/4fb3bf02-62ad-41af-8158-8f283d76e5c5/volumes" Jul 17 18:48:31.301851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:31.301816 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:48:54.092490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:48:54.092396 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:49:14.729359 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:49:14.729324 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:49:25.683612 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:49:25.683576 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:49:44.410433 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:49:44.410404 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:49:54.178650 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:49:54.178081 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:49:54.186368 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:49:54.186348 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:50:10.488690 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:10.488652 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:50:30.782735 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:30.782700 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:50:42.292691 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:42.292653 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:50:57.958706 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.958672 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:50:57.959104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959076 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088f4624-e805-498f-b26f-ecac02d92f4b" containerName="main" Jul 17 18:50:57.959104 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959088 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="088f4624-e805-498f-b26f-ecac02d92f4b" containerName="main" Jul 17 18:50:57.959209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959107 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" containerName="main" Jul 17 18:50:57.959209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959113 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" containerName="main" Jul 17 18:50:57.959209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959195 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb3bf02-62ad-41af-8158-8f283d76e5c5" containerName="main" Jul 17 18:50:57.959209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.959204 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="088f4624-e805-498f-b26f-ecac02d92f4b" containerName="main" Jul 17 18:50:57.962229 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.962213 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:57.964325 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.964292 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"kube-root-ca.crt\"" Jul 17 18:50:57.964814 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.964794 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"openshift-service-ca.crt\"" Jul 17 18:50:57.964925 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.964822 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"default-dockercfg-gs2pn\"" Jul 17 18:50:57.964925 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.964850 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"auth-test-model-4280a6-kserve-self-signed-certs\"" Jul 17 18:50:57.971319 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:57.971299 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:50:58.078033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078000 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.078033 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078035 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzstl\" (UniqueName: \"kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.078282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078117 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.078282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078216 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.078282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078252 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.078282 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.078275 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179175 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179143 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179175 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179175 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzstl\" (UniqueName: \"kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179421 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179204 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179421 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179253 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179421 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179275 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179421 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179289 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179731 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179704 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179807 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179749 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.179807 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.179770 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.181548 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.181517 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.181783 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.181766 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.186750 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.186729 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzstl\" (UniqueName: \"kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl\") pod \"auth-test-model-4280a6-kserve-777c8bb65f-fdjgm\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.272098 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.272069 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:50:58.402216 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:58.402190 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:50:58.404595 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:50:58.404567 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e977d4_56e4_47f1_9192_9240dbc83168.slice/crio-77fb5a7d98f81aaa8616130f4213af107e6060c66d2a7d3eb5671b6a6d2172b8 WatchSource:0}: Error finding container 77fb5a7d98f81aaa8616130f4213af107e6060c66d2a7d3eb5671b6a6d2172b8: Status 404 returned error can't find the container with id 77fb5a7d98f81aaa8616130f4213af107e6060c66d2a7d3eb5671b6a6d2172b8 Jul 17 18:50:59.319903 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:59.319866 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" event={"ID":"d9e977d4-56e4-47f1-9192-9240dbc83168","Type":"ContainerStarted","Data":"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96"} Jul 17 18:50:59.319903 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:59.319907 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" event={"ID":"d9e977d4-56e4-47f1-9192-9240dbc83168","Type":"ContainerStarted","Data":"77fb5a7d98f81aaa8616130f4213af107e6060c66d2a7d3eb5671b6a6d2172b8"} Jul 17 18:50:59.339557 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:50:59.339506 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" podStartSLOduration=2.339490634 podStartE2EDuration="2.339490634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:50:59.337938137 +0000 UTC m=+2466.156709400" watchObservedRunningTime="2026-07-17 18:50:59.339490634 +0000 UTC m=+2466.158261899" Jul 17 18:51:01.378152 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:01.378121 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:51:08.272854 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:08.272808 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:51:08.273351 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:08.272893 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:51:08.285209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:08.285179 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:51:08.371159 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:08.371127 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:51:25.757723 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.757690 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:51:25.761446 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.761429 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.763415 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.763393 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"default-dockercfg-vpnhx\"" Jul 17 18:51:25.764260 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.764231 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"kube-root-ca.crt\"" Jul 17 18:51:25.764374 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.764282 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"openshift-service-ca.crt\"" Jul 17 18:51:25.764441 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.764388 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"auth-test-model-1671a0-kserve-self-signed-certs\"" Jul 17 18:51:25.769846 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.769822 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:51:25.920751 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920709 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.920950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920761 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.920950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920794 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.920950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920817 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.920950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920850 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:25.920950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:25.920897 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxt7\" (UniqueName: \"kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021735 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021626 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021735 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021684 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxt7\" (UniqueName: \"kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021748 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021774 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021791 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.021968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.021815 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.022225 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.022177 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.022225 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.022188 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.022299 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.022273 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.023934 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.023911 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.024285 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.024255 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.030588 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.030568 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxt7\" (UniqueName: \"kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7\") pod \"auth-test-model-1671a0-kserve-78dc68555f-qqz22\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.071632 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.071600 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:26.199726 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.199702 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:51:26.201921 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:51:26.201892 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64da5b40_a4aa_434c_a89d_11672b5180fa.slice/crio-5fba7c0e6d9f8ef87d403379f34c04fe6ac202052298421b44582e7fa8eb5e50 WatchSource:0}: Error finding container 5fba7c0e6d9f8ef87d403379f34c04fe6ac202052298421b44582e7fa8eb5e50: Status 404 returned error can't find the container with id 5fba7c0e6d9f8ef87d403379f34c04fe6ac202052298421b44582e7fa8eb5e50 Jul 17 18:51:26.443592 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.443512 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" event={"ID":"64da5b40-a4aa-434c-a89d-11672b5180fa","Type":"ContainerStarted","Data":"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7"} Jul 17 18:51:26.443592 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.443554 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" event={"ID":"64da5b40-a4aa-434c-a89d-11672b5180fa","Type":"ContainerStarted","Data":"5fba7c0e6d9f8ef87d403379f34c04fe6ac202052298421b44582e7fa8eb5e50"} Jul 17 18:51:26.459452 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:26.459408 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" podStartSLOduration=1.459393145 podStartE2EDuration="1.459393145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:26.458077049 +0000 UTC m=+2493.276848312" watchObservedRunningTime="2026-07-17 18:51:26.459393145 +0000 UTC m=+2493.278164408" Jul 17 18:51:29.571994 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:29.571942 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:51:36.072249 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:36.072198 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:36.072625 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:36.072266 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:36.084262 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:36.084228 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:36.500425 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:36.500346 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:51:45.577544 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:51:45.577455 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:52:04.469466 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:52:04.469431 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:52:23.180838 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:52:23.180803 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:52:37.878021 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:52:37.877972 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:52:53.579199 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:52:53.579164 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:53:12.072100 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:12.072061 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:53:26.422897 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:26.422864 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:53:27.469043 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:27.469009 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:53:30.179320 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:30.179280 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:53:41.178186 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.178146 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:53:41.181657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.181635 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.184255 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.184217 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"sub-test-model-4280a6-kserve-self-signed-certs\"" Jul 17 18:53:41.196976 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.196954 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:53:41.263121 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263083 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.263298 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263138 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.263298 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263223 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.263298 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263249 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.263298 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263285 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.263443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.263384 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2s5\" (UniqueName: \"kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364443 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364384 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364452 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364499 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364540 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2s5\" (UniqueName: \"kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364600 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364581 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364799 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364619 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364876 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364851 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364933 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364890 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.364933 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.364911 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.366842 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.366815 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.367122 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.367105 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.373360 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.373344 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2s5\" (UniqueName: \"kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5\") pod \"sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.442730 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.442628 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" podUID="d9e977d4-56e4-47f1-9192-9240dbc83168" containerName="main" containerID="cri-o://e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96" gracePeriod=45 Jul 17 18:53:41.491657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.491624 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:41.635135 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.635107 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:53:41.637197 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:53:41.637165 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197402db_3852_4fd4_b6df_1d246867b205.slice/crio-7d709bb444d6541f665a9679d8ce558a2c06def1c3b1812c4ca853a6d1c78a52 WatchSource:0}: Error finding container 7d709bb444d6541f665a9679d8ce558a2c06def1c3b1812c4ca853a6d1c78a52: Status 404 returned error can't find the container with id 7d709bb444d6541f665a9679d8ce558a2c06def1c3b1812c4ca853a6d1c78a52 Jul 17 18:53:41.686698 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.686678 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:53:41.869431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869396 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869628 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869462 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzstl\" (UniqueName: \"kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869628 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869491 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869628 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869525 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869628 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869573 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869628 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869616 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm\") pod \"d9e977d4-56e4-47f1-9192-9240dbc83168\" (UID: \"d9e977d4-56e4-47f1-9192-9240dbc83168\") " Jul 17 18:53:41.869879 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869745 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:41.869879 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869829 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:41.869879 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.869862 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home" (OuterVolumeSpecName: "home") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:41.870127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.870109 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:41.870193 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.870131 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:41.870193 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.870144 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:41.871896 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.871863 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl" (OuterVolumeSpecName: "kube-api-access-hzstl") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "kube-api-access-hzstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:53:41.871896 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.871877 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:53:41.871896 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.871882 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm" (OuterVolumeSpecName: "dshm") pod "d9e977d4-56e4-47f1-9192-9240dbc83168" (UID: "d9e977d4-56e4-47f1-9192-9240dbc83168"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:41.971709 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.971605 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e977d4-56e4-47f1-9192-9240dbc83168-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:41.971709 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.971674 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzstl\" (UniqueName: \"kubernetes.io/projected/d9e977d4-56e4-47f1-9192-9240dbc83168-kube-api-access-hzstl\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:41.971709 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:41.971690 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e977d4-56e4-47f1-9192-9240dbc83168-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.082392 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.082354 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" event={"ID":"197402db-3852-4fd4-b6df-1d246867b205","Type":"ContainerStarted","Data":"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6"} Jul 17 18:53:42.082612 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.082401 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" event={"ID":"197402db-3852-4fd4-b6df-1d246867b205","Type":"ContainerStarted","Data":"7d709bb444d6541f665a9679d8ce558a2c06def1c3b1812c4ca853a6d1c78a52"} Jul 17 18:53:42.083620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.083597 2587 generic.go:356] "Generic (PLEG): container finished" podID="d9e977d4-56e4-47f1-9192-9240dbc83168" containerID="e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96" exitCode=0 Jul 17 18:53:42.083705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.083629 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" event={"ID":"d9e977d4-56e4-47f1-9192-9240dbc83168","Type":"ContainerDied","Data":"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96"} Jul 17 18:53:42.083705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.083642 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" Jul 17 18:53:42.083705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.083649 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm" event={"ID":"d9e977d4-56e4-47f1-9192-9240dbc83168","Type":"ContainerDied","Data":"77fb5a7d98f81aaa8616130f4213af107e6060c66d2a7d3eb5671b6a6d2172b8"} Jul 17 18:53:42.083705 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.083665 2587 scope.go:117] "RemoveContainer" containerID="e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96" Jul 17 18:53:42.092272 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.092254 2587 scope.go:117] "RemoveContainer" containerID="e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96" Jul 17 18:53:42.092527 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:53:42.092508 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96\": container with ID starting with e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96 not found: ID does not exist" containerID="e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96" Jul 17 18:53:42.092584 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.092538 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96"} err="failed to get container status \"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96\": rpc error: code = NotFound desc = could not find container \"e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96\": container with ID starting with e30a963031b1ecd549dd88be0adc2b88391697c99f155c197fed262835baaf96 not found: ID does not exist" Jul 17 18:53:42.104089 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.104045 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" podStartSLOduration=1.104030543 podStartE2EDuration="1.104030543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:53:42.102215325 +0000 UTC m=+2628.920986591" watchObservedRunningTime="2026-07-17 18:53:42.104030543 +0000 UTC m=+2628.922801806" Jul 17 18:53:42.116014 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.115972 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:53:42.121873 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.121851 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/auth-test-model-4280a6-kserve-777c8bb65f-fdjgm"] Jul 17 18:53:42.489177 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.489131 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" podUID="64da5b40-a4aa-434c-a89d-11672b5180fa" containerName="main" containerID="cri-o://3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7" gracePeriod=45 Jul 17 18:53:42.739279 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.739251 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:53:42.880219 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880192 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880219 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880226 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880444 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880313 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880444 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880352 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880444 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880409 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880598 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880464 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hxt7\" (UniqueName: \"kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7\") pod \"64da5b40-a4aa-434c-a89d-11672b5180fa\" (UID: \"64da5b40-a4aa-434c-a89d-11672b5180fa\") " Jul 17 18:53:42.880598 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880569 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:42.880598 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880581 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home" (OuterVolumeSpecName: "home") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:42.880747 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880730 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:42.880813 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880797 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.881067 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.880819 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.882517 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.882494 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm" (OuterVolumeSpecName: "dshm") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:53:42.882517 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.882508 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:53:42.882648 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.882576 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7" (OuterVolumeSpecName: "kube-api-access-7hxt7") pod "64da5b40-a4aa-434c-a89d-11672b5180fa" (UID: "64da5b40-a4aa-434c-a89d-11672b5180fa"). InnerVolumeSpecName "kube-api-access-7hxt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:53:42.982101 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.982015 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxt7\" (UniqueName: \"kubernetes.io/projected/64da5b40-a4aa-434c-a89d-11672b5180fa-kube-api-access-7hxt7\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.982101 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.982050 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.982101 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.982065 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da5b40-a4aa-434c-a89d-11672b5180fa-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:42.982101 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:42.982076 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da5b40-a4aa-434c-a89d-11672b5180fa-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:53:43.091349 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.091309 2587 generic.go:356] "Generic (PLEG): container finished" podID="64da5b40-a4aa-434c-a89d-11672b5180fa" containerID="3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7" exitCode=0 Jul 17 18:53:43.091512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.091356 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" Jul 17 18:53:43.091512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.091392 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" event={"ID":"64da5b40-a4aa-434c-a89d-11672b5180fa","Type":"ContainerDied","Data":"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7"} Jul 17 18:53:43.091512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.091440 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22" event={"ID":"64da5b40-a4aa-434c-a89d-11672b5180fa","Type":"ContainerDied","Data":"5fba7c0e6d9f8ef87d403379f34c04fe6ac202052298421b44582e7fa8eb5e50"} Jul 17 18:53:43.091512 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.091461 2587 scope.go:117] "RemoveContainer" containerID="3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7" Jul 17 18:53:43.100227 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.100212 2587 scope.go:117] "RemoveContainer" containerID="3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7" Jul 17 18:53:43.100458 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:53:43.100442 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7\": container with ID starting with 3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7 not found: ID does not exist" containerID="3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7" Jul 17 18:53:43.100508 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.100465 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7"} err="failed to get container status \"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7\": rpc error: code = NotFound desc = could not find container \"3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7\": container with ID starting with 3b8f62d5024e25b30c71315cae55d69a0256071983a9866c2efe981db85cd0c7 not found: ID does not exist" Jul 17 18:53:43.112767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.112747 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:53:43.116843 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.116822 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/auth-test-model-1671a0-kserve-78dc68555f-qqz22"] Jul 17 18:53:43.740356 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.740325 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64da5b40-a4aa-434c-a89d-11672b5180fa" path="/var/lib/kubelet/pods/64da5b40-a4aa-434c-a89d-11672b5180fa/volumes" Jul 17 18:53:43.740714 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:43.740698 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e977d4-56e4-47f1-9192-9240dbc83168" path="/var/lib/kubelet/pods/d9e977d4-56e4-47f1-9192-9240dbc83168/volumes" Jul 17 18:53:46.175960 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:46.175917 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:53:51.492057 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:51.492017 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:51.492446 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:51.492070 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:51.504458 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:51.504429 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:53:52.149017 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:53:52.148976 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:54:03.486040 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486002 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:54:03.486490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486391 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e977d4-56e4-47f1-9192-9240dbc83168" containerName="main" Jul 17 18:54:03.486490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486405 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e977d4-56e4-47f1-9192-9240dbc83168" containerName="main" Jul 17 18:54:03.486490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486422 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64da5b40-a4aa-434c-a89d-11672b5180fa" containerName="main" Jul 17 18:54:03.486490 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486429 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="64da5b40-a4aa-434c-a89d-11672b5180fa" containerName="main" Jul 17 18:54:03.486620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486495 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="64da5b40-a4aa-434c-a89d-11672b5180fa" containerName="main" Jul 17 18:54:03.486620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.486526 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e977d4-56e4-47f1-9192-9240dbc83168" containerName="main" Jul 17 18:54:03.489429 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.489412 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.492662 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.492635 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"default-dockercfg-vpnhx\"" Jul 17 18:54:03.493116 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.493095 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"openshift-service-ca.crt\"" Jul 17 18:54:03.493349 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.493324 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"sub-test-model-1671a0-kserve-self-signed-certs\"" Jul 17 18:54:03.493523 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.493502 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"kube-root-ca.crt\"" Jul 17 18:54:03.501799 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.501772 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:54:03.570808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570764 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.570808 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570812 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqjx\" (UniqueName: \"kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.571065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570877 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.571065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570906 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.571065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570936 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.571065 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.570960 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.671867 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.671834 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.671875 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqjx\" (UniqueName: \"kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.671920 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.671951 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.672008 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.672040 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672406 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.672387 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672475 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.672433 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.672475 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.672451 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.674177 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.674155 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.674461 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.674442 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.680110 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.680089 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqjx\" (UniqueName: \"kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx\") pod \"sub-test-model-1671a0-kserve-754fc48f47-5kfbz\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.803166 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.803126 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:03.937181 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:03.937150 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:54:03.940294 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:54:03.940266 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa82e5b_cd7c_46cf_8e7b_5c7578e447b2.slice/crio-bab8e794a703645377b6021cbebb929794af95ebee9ba29be45fd02fee27cb89 WatchSource:0}: Error finding container bab8e794a703645377b6021cbebb929794af95ebee9ba29be45fd02fee27cb89: Status 404 returned error can't find the container with id bab8e794a703645377b6021cbebb929794af95ebee9ba29be45fd02fee27cb89 Jul 17 18:54:04.197025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:04.196914 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" event={"ID":"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2","Type":"ContainerStarted","Data":"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93"} Jul 17 18:54:04.197025 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:04.196956 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" event={"ID":"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2","Type":"ContainerStarted","Data":"bab8e794a703645377b6021cbebb929794af95ebee9ba29be45fd02fee27cb89"} Jul 17 18:54:04.215668 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:04.215611 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" podStartSLOduration=1.215595459 podStartE2EDuration="1.215595459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:04.214906683 +0000 UTC m=+2651.033677947" watchObservedRunningTime="2026-07-17 18:54:04.215595459 +0000 UTC m=+2651.034366722" Jul 17 18:54:08.485611 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:08.485572 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:54:13.803865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:13.803823 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:13.804386 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:13.804153 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:13.816725 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:13.816695 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:14.256932 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:14.256906 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:54:23.276937 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:23.276899 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:54:43.191330 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:43.191285 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:54:48.894451 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:48.894416 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:54:49.880460 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:49.880429 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:54:54.208797 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:54.208688 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:54:54.216140 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:54.216119 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ljz5r_3fbdc11d-1db8-47f7-8675-f5c1b9ff6158/ovn-acl-logging/0.log" Jul 17 18:54:59.592851 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.590184 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:54:59.594942 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.594912 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.597416 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.597397 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4280a6\"/\"rate-test-model-4280a6-kserve-self-signed-certs\"" Jul 17 18:54:59.601352 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.601329 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:54:59.700604 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700569 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.700604 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700602 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.700817 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700632 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.700817 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700720 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.700817 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700771 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.700924 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.700865 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7sc\" (UniqueName: \"kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.801955 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.801921 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.801955 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.801956 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802179 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802018 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802179 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802049 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802179 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802095 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802179 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802174 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7sc\" (UniqueName: \"kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802479 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802450 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802532 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.802605 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.802537 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.804322 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.804296 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.804501 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.804484 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.810912 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.810895 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7sc\" (UniqueName: \"kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc\") pod \"rate-test-model-4280a6-kserve-59c76c9767-tjf2v\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:54:59.904910 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:54:59.904830 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:55:00.034278 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:00.034244 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:55:00.036968 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:55:00.036934 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82f50599_cdd4_47fa_9a14_5e429da5424f.slice/crio-354ba4568782c2ec835603778b1abcd6b0300129d188a774459ac839b95104af WatchSource:0}: Error finding container 354ba4568782c2ec835603778b1abcd6b0300129d188a774459ac839b95104af: Status 404 returned error can't find the container with id 354ba4568782c2ec835603778b1abcd6b0300129d188a774459ac839b95104af Jul 17 18:55:00.459681 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:00.459644 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" event={"ID":"82f50599-cdd4-47fa-9a14-5e429da5424f","Type":"ContainerStarted","Data":"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece"} Jul 17 18:55:00.459681 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:00.459685 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" event={"ID":"82f50599-cdd4-47fa-9a14-5e429da5424f","Type":"ContainerStarted","Data":"354ba4568782c2ec835603778b1abcd6b0300129d188a774459ac839b95104af"} Jul 17 18:55:00.482281 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:00.482234 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" podStartSLOduration=1.48222057 podStartE2EDuration="1.48222057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:00.480784499 +0000 UTC m=+2707.299555765" watchObservedRunningTime="2026-07-17 18:55:00.48222057 +0000 UTC m=+2707.300991831" Jul 17 18:55:03.185197 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:03.185167 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:55:03.914178 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:03.914121 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" podUID="197402db-3852-4fd4-b6df-1d246867b205" containerName="main" containerID="cri-o://8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6" gracePeriod=45 Jul 17 18:55:04.170444 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.170388 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:55:04.343001 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.342959 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343092 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343113 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343128 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343152 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343246 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v2s5\" (UniqueName: \"kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5\") pod \"197402db-3852-4fd4-b6df-1d246867b205\" (UID: \"197402db-3852-4fd4-b6df-1d246867b205\") " Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343392 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache" (OuterVolumeSpecName: "model-cache") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343413 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:04.343525 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343491 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home" (OuterVolumeSpecName: "home") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:04.343950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343637 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.343950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343657 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.343950 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.343670 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.345299 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.345273 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm" (OuterVolumeSpecName: "dshm") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:04.345436 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.345419 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5" (OuterVolumeSpecName: "kube-api-access-9v2s5") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "kube-api-access-9v2s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:55:04.345551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.345535 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "197402db-3852-4fd4-b6df-1d246867b205" (UID: "197402db-3852-4fd4-b6df-1d246867b205"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:55:04.444286 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.444184 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v2s5\" (UniqueName: \"kubernetes.io/projected/197402db-3852-4fd4-b6df-1d246867b205-kube-api-access-9v2s5\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.444286 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.444215 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197402db-3852-4fd4-b6df-1d246867b205-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.444286 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.444238 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197402db-3852-4fd4-b6df-1d246867b205-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.478250 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.478220 2587 generic.go:356] "Generic (PLEG): container finished" podID="197402db-3852-4fd4-b6df-1d246867b205" containerID="8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6" exitCode=0 Jul 17 18:55:04.478379 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.478269 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" Jul 17 18:55:04.478379 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.478269 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" event={"ID":"197402db-3852-4fd4-b6df-1d246867b205","Type":"ContainerDied","Data":"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6"} Jul 17 18:55:04.478447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.478386 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6" event={"ID":"197402db-3852-4fd4-b6df-1d246867b205","Type":"ContainerDied","Data":"7d709bb444d6541f665a9679d8ce558a2c06def1c3b1812c4ca853a6d1c78a52"} Jul 17 18:55:04.478478 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.478414 2587 scope.go:117] "RemoveContainer" containerID="8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6" Jul 17 18:55:04.489561 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.488754 2587 scope.go:117] "RemoveContainer" containerID="8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6" Jul 17 18:55:04.490040 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:55:04.490003 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6\": container with ID starting with 8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6 not found: ID does not exist" containerID="8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6" Jul 17 18:55:04.490147 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.490054 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6"} err="failed to get container status \"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6\": rpc error: code = NotFound desc = could not find container \"8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6\": container with ID starting with 8f7102b7f591fb77bef54877dd9b15e8790e614b2effeea97db5442a5650bcd6 not found: ID does not exist" Jul 17 18:55:04.501447 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.501422 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:55:04.505445 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.505425 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/sub-test-model-4280a6-kserve-6bbb8675f7-v9sc6"] Jul 17 18:55:04.900708 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:04.900667 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" podUID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" containerName="main" containerID="cri-o://3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93" gracePeriod=45 Jul 17 18:55:05.153348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.153280 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:55:05.250616 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250583 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpqjx\" (UniqueName: \"kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.250616 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250621 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.250865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250641 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.250865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250689 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.250865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250707 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.250865 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250739 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm\") pod \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\" (UID: \"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2\") " Jul 17 18:55:05.251134 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.250900 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:05.251134 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.251031 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:05.251134 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.251037 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.251134 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.251070 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home" (OuterVolumeSpecName: "home") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:05.252816 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.252792 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm" (OuterVolumeSpecName: "dshm") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:05.252890 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.252844 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:55:05.252959 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.252938 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx" (OuterVolumeSpecName: "kube-api-access-xpqjx") pod "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" (UID: "3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2"). InnerVolumeSpecName "kube-api-access-xpqjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:55:05.351597 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.351554 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqjx\" (UniqueName: \"kubernetes.io/projected/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-kube-api-access-xpqjx\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.351597 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.351588 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.351597 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.351597 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.351597 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.351605 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.352112 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.351612 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:55:05.483881 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.483786 2587 generic.go:356] "Generic (PLEG): container finished" podID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" containerID="3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93" exitCode=0 Jul 17 18:55:05.483881 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.483834 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" Jul 17 18:55:05.484115 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.483876 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" event={"ID":"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2","Type":"ContainerDied","Data":"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93"} Jul 17 18:55:05.484115 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.483927 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz" event={"ID":"3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2","Type":"ContainerDied","Data":"bab8e794a703645377b6021cbebb929794af95ebee9ba29be45fd02fee27cb89"} Jul 17 18:55:05.484115 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.483953 2587 scope.go:117] "RemoveContainer" containerID="3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93" Jul 17 18:55:05.493112 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.493094 2587 scope.go:117] "RemoveContainer" containerID="3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93" Jul 17 18:55:05.493393 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:55:05.493375 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93\": container with ID starting with 3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93 not found: ID does not exist" containerID="3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93" Jul 17 18:55:05.493457 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.493403 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93"} err="failed to get container status \"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93\": rpc error: code = NotFound desc = could not find container \"3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93\": container with ID starting with 3e14eae4f6dbae3bd63cedbcc2adb4a43803878703883cf1191e788f4fb33b93 not found: ID does not exist" Jul 17 18:55:05.503592 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.503566 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:55:05.508267 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.508234 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/sub-test-model-1671a0-kserve-754fc48f47-5kfbz"] Jul 17 18:55:05.740287 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.740196 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197402db-3852-4fd4-b6df-1d246867b205" path="/var/lib/kubelet/pods/197402db-3852-4fd4-b6df-1d246867b205/volumes" Jul 17 18:55:05.740613 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:05.740599 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" path="/var/lib/kubelet/pods/3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2/volumes" Jul 17 18:55:09.906052 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:09.906016 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:55:09.906645 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:09.906071 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:55:09.918431 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:09.918408 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:55:10.519334 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:10.519306 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:55:19.071209 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071176 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:55:19.071607 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071570 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197402db-3852-4fd4-b6df-1d246867b205" containerName="main" Jul 17 18:55:19.071607 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071581 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="197402db-3852-4fd4-b6df-1d246867b205" containerName="main" Jul 17 18:55:19.071607 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071605 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" containerName="main" Jul 17 18:55:19.071607 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071610 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" containerName="main" Jul 17 18:55:19.071767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071671 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa82e5b-cd7c-46cf-8e7b-5c7578e447b2" containerName="main" Jul 17 18:55:19.071767 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.071682 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="197402db-3852-4fd4-b6df-1d246867b205" containerName="main" Jul 17 18:55:19.076061 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.076039 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.078938 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.078819 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"kube-root-ca.crt\"" Jul 17 18:55:19.078938 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.078824 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"openshift-service-ca.crt\"" Jul 17 18:55:19.078938 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.078909 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"rate-test-model-1671a0-kserve-self-signed-certs\"" Jul 17 18:55:19.078938 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.078913 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1671a0\"/\"default-dockercfg-vpnhx\"" Jul 17 18:55:19.088902 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.088874 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:55:19.180402 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180360 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.180402 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180401 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.180657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180478 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.180657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180495 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tlk\" (UniqueName: \"kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.180657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180549 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.180657 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.180593 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281518 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281554 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281554 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281597 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281612 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tlk\" (UniqueName: \"kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281641 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.281785 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281714 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.282013 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281932 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.282013 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.281977 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.282127 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.282077 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.283852 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.283833 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.284148 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.284129 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.288880 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.288857 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tlk\" (UniqueName: \"kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk\") pod \"rate-test-model-1671a0-kserve-dcdf7b579-fntq5\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.386794 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.386708 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:19.517877 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.517845 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:55:19.521176 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:55:19.521151 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81ee6532_000e_493a_a45a_02a4326beede.slice/crio-98a3eaa22a072d00ee24b5c074d95db77b061d1af30121880dc4e94699fb5240 WatchSource:0}: Error finding container 98a3eaa22a072d00ee24b5c074d95db77b061d1af30121880dc4e94699fb5240: Status 404 returned error can't find the container with id 98a3eaa22a072d00ee24b5c074d95db77b061d1af30121880dc4e94699fb5240 Jul 17 18:55:19.550562 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:19.550534 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" event={"ID":"81ee6532-000e-493a-a45a-02a4326beede","Type":"ContainerStarted","Data":"98a3eaa22a072d00ee24b5c074d95db77b061d1af30121880dc4e94699fb5240"} Jul 17 18:55:20.556771 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:20.556739 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" event={"ID":"81ee6532-000e-493a-a45a-02a4326beede","Type":"ContainerStarted","Data":"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade"} Jul 17 18:55:20.572861 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:20.572806 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" podStartSLOduration=1.572785853 podStartE2EDuration="1.572785853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:20.572506942 +0000 UTC m=+2727.391278262" watchObservedRunningTime="2026-07-17 18:55:20.572785853 +0000 UTC m=+2727.391557115" Jul 17 18:55:25.281930 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:25.281897 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:55:29.387772 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:29.387742 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:29.387772 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:29.387778 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:29.400123 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:29.400102 2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:29.608566 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:29.608539 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:55:51.982066 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:55:51.982029 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:56:06.976176 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:06.976136 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:56:13.427024 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:13.426976 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:56:14.581688 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:14.581656 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:56:21.071458 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:21.071425 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:56:28.446968 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.446909 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" podUID="82f50599-cdd4-47fa-9a14-5e429da5424f" containerName="main" containerID="cri-o://d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece" gracePeriod=45 Jul 17 18:56:28.703276 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.703216 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:56:28.803260 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803222 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803260 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803265 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803513 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803310 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803513 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803362 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803513 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803473 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache" (OuterVolumeSpecName: "model-cache") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:28.803513 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803498 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h7sc\" (UniqueName: \"kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803702 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803549 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm\") pod \"82f50599-cdd4-47fa-9a14-5e429da5424f\" (UID: \"82f50599-cdd4-47fa-9a14-5e429da5424f\") " Jul 17 18:56:28.803702 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803623 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:28.803702 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803636 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home" (OuterVolumeSpecName: "home") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:28.803860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803798 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.803860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803809 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.803860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.803820 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.805557 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.805530 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm" (OuterVolumeSpecName: "dshm") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:28.805557 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.805537 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:56:28.805704 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.805578 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc" (OuterVolumeSpecName: "kube-api-access-4h7sc") pod "82f50599-cdd4-47fa-9a14-5e429da5424f" (UID: "82f50599-cdd4-47fa-9a14-5e429da5424f"). InnerVolumeSpecName "kube-api-access-4h7sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:28.877182 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.877150 2587 generic.go:356] "Generic (PLEG): container finished" podID="82f50599-cdd4-47fa-9a14-5e429da5424f" containerID="d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece" exitCode=0 Jul 17 18:56:28.877348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.877190 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" Jul 17 18:56:28.877348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.877194 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" event={"ID":"82f50599-cdd4-47fa-9a14-5e429da5424f","Type":"ContainerDied","Data":"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece"} Jul 17 18:56:28.877348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.877285 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v" event={"ID":"82f50599-cdd4-47fa-9a14-5e429da5424f","Type":"ContainerDied","Data":"354ba4568782c2ec835603778b1abcd6b0300129d188a774459ac839b95104af"} Jul 17 18:56:28.877348 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.877301 2587 scope.go:117] "RemoveContainer" containerID="d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece" Jul 17 18:56:28.886539 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.886523 2587 scope.go:117] "RemoveContainer" containerID="d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece" Jul 17 18:56:28.886776 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:56:28.886757 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece\": container with ID starting with d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece not found: ID does not exist" containerID="d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece" Jul 17 18:56:28.886840 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.886786 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece"} err="failed to get container status \"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece\": rpc error: code = NotFound desc = could not find container \"d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece\": container with ID starting with d22685bed8f20a85058c8d7105976031cb8a4a6b718b9cbb4c270d6b1ede4ece not found: ID does not exist" Jul 17 18:56:28.899167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.899141 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:56:28.904401 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.904381 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h7sc\" (UniqueName: \"kubernetes.io/projected/82f50599-cdd4-47fa-9a14-5e429da5424f-kube-api-access-4h7sc\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.904401 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.904403 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82f50599-cdd4-47fa-9a14-5e429da5424f-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.904551 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.904413 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82f50599-cdd4-47fa-9a14-5e429da5424f-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:28.904913 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:28.904892 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4280a6/rate-test-model-4280a6-kserve-59c76c9767-tjf2v"] Jul 17 18:56:29.601729 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.601683 2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" podUID="81ee6532-000e-493a-a45a-02a4326beede" containerName="main" containerID="cri-o://ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade" gracePeriod=45 Jul 17 18:56:29.606379 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.606337 2587 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" podUID="81ee6532-000e-493a-a45a-02a4326beede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": EOF" Jul 17 18:56:29.741056 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.741023 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f50599-cdd4-47fa-9a14-5e429da5424f" path="/var/lib/kubelet/pods/82f50599-cdd4-47fa-9a14-5e429da5424f/volumes" Jul 17 18:56:29.858956 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.858889 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:56:29.883148 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.883110 2587 generic.go:356] "Generic (PLEG): container finished" podID="81ee6532-000e-493a-a45a-02a4326beede" containerID="ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade" exitCode=0 Jul 17 18:56:29.883148 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.883151 2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" Jul 17 18:56:29.883397 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.883190 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" event={"ID":"81ee6532-000e-493a-a45a-02a4326beede","Type":"ContainerDied","Data":"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade"} Jul 17 18:56:29.883397 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.883236 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5" event={"ID":"81ee6532-000e-493a-a45a-02a4326beede","Type":"ContainerDied","Data":"98a3eaa22a072d00ee24b5c074d95db77b061d1af30121880dc4e94699fb5240"} Jul 17 18:56:29.883397 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.883258 2587 scope.go:117] "RemoveContainer" containerID="ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade" Jul 17 18:56:29.891890 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.891872 2587 scope.go:117] "RemoveContainer" containerID="ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade" Jul 17 18:56:29.892174 ip-10-0-136-225 kubenswrapper[2587]: E0717 18:56:29.892152 2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade\": container with ID starting with ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade not found: ID does not exist" containerID="ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade" Jul 17 18:56:29.892244 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:29.892183 2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade"} err="failed to get container status \"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade\": rpc error: code = NotFound desc = could not find container \"ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade\": container with ID starting with ead4b1fe8a6f08654d79e7d44bce411ccb6ae7d26c8cb4627b7ea8a6f3632ade not found: ID does not exist" Jul 17 18:56:30.015121 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015077 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015121 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015120 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015427 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015185 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7tlk\" (UniqueName: \"kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015427 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015223 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015427 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015263 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015427 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015303 2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache\") pod \"81ee6532-000e-493a-a45a-02a4326beede\" (UID: \"81ee6532-000e-493a-a45a-02a4326beede\") " Jul 17 18:56:30.015622 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015445 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home" (OuterVolumeSpecName: "home") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:30.015622 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015576 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache" (OuterVolumeSpecName: "model-cache") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:30.015622 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015518 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:30.015769 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015654 2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-tmp-dir\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.015769 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015673 2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-model-cache\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.015769 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.015681 2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-home\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.017435 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.017408 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:56:30.017435 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.017423 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk" (OuterVolumeSpecName: "kube-api-access-q7tlk") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "kube-api-access-q7tlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:30.017435 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.017424 2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm" (OuterVolumeSpecName: "dshm") pod "81ee6532-000e-493a-a45a-02a4326beede" (UID: "81ee6532-000e-493a-a45a-02a4326beede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:30.116789 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.116680 2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ee6532-000e-493a-a45a-02a4326beede-tls-certs\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.116789 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.116728 2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ee6532-000e-493a-a45a-02a4326beede-dshm\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.116789 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.116738 2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tlk\" (UniqueName: \"kubernetes.io/projected/81ee6532-000e-493a-a45a-02a4326beede-kube-api-access-q7tlk\") on node \"ip-10-0-136-225.ec2.internal\" DevicePath \"\"" Jul 17 18:56:30.203741 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.203709 2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:56:30.207779 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:30.207752 2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1671a0/rate-test-model-1671a0-kserve-dcdf7b579-fntq5"] Jul 17 18:56:31.739690 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:31.739659 2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ee6532-000e-493a-a45a-02a4326beede" path="/var/lib/kubelet/pods/81ee6532-000e-493a-a45a-02a4326beede/volumes" Jul 17 18:56:47.475824 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:56:47.475789 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:57:13.375285 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:13.375250 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4xqw9"] Jul 17 18:57:32.518727 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:32.518698 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-99g9h_7028b7bf-1c02-4809-8879-ef8088f88e68/copy-manifests/0.log" Jul 17 18:57:32.534590 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:32.534565 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-99g9h_7028b7bf-1c02-4809-8879-ef8088f88e68/manager/0.log" Jul 17 18:57:36.543184 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:36.543156 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wqh75_76efa28c-2bd9-465f-9afd-40272685ebaf/kuadrant-console-plugin/0.log" Jul 17 18:57:36.896888 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:36.896856 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4xqw9_fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa/limitador/0.log" Jul 17 18:57:37.730310 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:37.730281 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-vwtnq_00f83132-f55f-42d9-91d0-e95d5e4af89e/kube-auth-proxy/0.log" Jul 17 18:57:38.782841 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:38.782765 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9dj26_e85d146a-83ff-49a3-9c73-3bda0f4f82de/main/0.log" Jul 17 18:57:39.015497 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:39.015467 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-g79jw_75e59e2d-85f9-41f9-b171-9ea6616b65c0/main/0.log" Jul 17 18:57:39.141929 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:39.141862 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cpvbd_a4d56e3b-71ce-4a02-8134-b8d08c6e6856/main/0.log" Jul 17 18:57:47.425674 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:47.425604 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4nfq7_73cf7878-9139-48a9-a8ea-5c958c28dd3d/global-pull-secret-syncer/0.log" Jul 17 18:57:47.652521 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:47.652492 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v5fsc_aad4f185-f146-4291-a953-ed2739ca5b82/konnectivity-agent/0.log" Jul 17 18:57:47.706156 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:47.706087 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-225.ec2.internal_c34073dec3e096eb00af0ee236229ab6/haproxy/0.log" Jul 17 18:57:53.162674 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:53.162646 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wqh75_76efa28c-2bd9-465f-9afd-40272685ebaf/kuadrant-console-plugin/0.log" Jul 17 18:57:53.424172 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:53.424077 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4xqw9_fede4f0f-ce6d-4702-9e4c-d48aa6d7d7aa/limitador/0.log" Jul 17 18:57:55.127198 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.127122 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/alertmanager/0.log" Jul 17 18:57:55.152577 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.152553 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/config-reloader/0.log" Jul 17 18:57:55.183475 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.183452 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/kube-rbac-proxy-web/0.log" Jul 17 18:57:55.207167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.207147 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/kube-rbac-proxy/0.log" Jul 17 18:57:55.231406 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.231385 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/kube-rbac-proxy-metric/0.log" Jul 17 18:57:55.255296 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.255272 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/prom-label-proxy/0.log" Jul 17 18:57:55.280620 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.280603 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_067c77b2-f82c-47a1-af81-d775c603f802/init-config-reloader/0.log" Jul 17 18:57:55.431381 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.431303 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c599bf6db-9h9cw_c171558a-c4c4-417d-bd00-9624f79cb9ae/metrics-server/0.log" Jul 17 18:57:55.460039 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.460009 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-t4fdw_b8af67c7-2a92-4f12-80a9-4663c3d6a19b/monitoring-plugin/0.log" Jul 17 18:57:55.501802 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.501776 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h5d2_740e8180-4df5-44d8-b136-1e5265fc0aa7/node-exporter/0.log" Jul 17 18:57:55.526146 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.526123 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h5d2_740e8180-4df5-44d8-b136-1e5265fc0aa7/kube-rbac-proxy/0.log" Jul 17 18:57:55.547944 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.547926 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h5d2_740e8180-4df5-44d8-b136-1e5265fc0aa7/init-textfile/0.log" Jul 17 18:57:55.733518 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.733449 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-p95fz_2d46514e-aa5b-4682-9b23-503119a62222/kube-rbac-proxy-main/0.log" Jul 17 18:57:55.762197 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.762168 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-p95fz_2d46514e-aa5b-4682-9b23-503119a62222/kube-rbac-proxy-self/0.log" Jul 17 18:57:55.791626 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:55.791604 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-p95fz_2d46514e-aa5b-4682-9b23-503119a62222/openshift-state-metrics/0.log" Jul 17 18:57:56.027167 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.027130 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-4jrs9_5998763a-0cce-4fcb-b486-ab98a59b2d74/prometheus-operator/0.log" Jul 17 18:57:56.048278 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.048256 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-4jrs9_5998763a-0cce-4fcb-b486-ab98a59b2d74/kube-rbac-proxy/0.log" Jul 17 18:57:56.075470 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.075449 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-b2v6l_aa6cbf55-c59e-4862-b10e-53f7d881f3ba/prometheus-operator-admission-webhook/0.log" Jul 17 18:57:56.108203 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.108186 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755bfb49b6-lzz5v_0f535c45-1476-4127-bd45-6b33e94405f1/telemeter-client/0.log" Jul 17 18:57:56.150717 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.150701 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755bfb49b6-lzz5v_0f535c45-1476-4127-bd45-6b33e94405f1/reload/0.log" Jul 17 18:57:56.177093 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.177074 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-755bfb49b6-lzz5v_0f535c45-1476-4127-bd45-6b33e94405f1/kube-rbac-proxy/0.log" Jul 17 18:57:56.208228 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.208207 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/thanos-query/0.log" Jul 17 18:57:56.233448 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.233430 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/kube-rbac-proxy-web/0.log" Jul 17 18:57:56.260401 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.260387 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/kube-rbac-proxy/0.log" Jul 17 18:57:56.281860 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.281782 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/prom-label-proxy/0.log" Jul 17 18:57:56.308454 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.308438 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/kube-rbac-proxy-rules/0.log" Jul 17 18:57:56.332794 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:56.332775 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-556b84bb4f-6zjqt_90ecdd24-7814-494b-8674-6ef55ae965e8/kube-rbac-proxy-metrics/0.log" Jul 17 18:57:57.250364 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.250315 2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg"] Jul 17 18:57:57.251232 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251207 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ee6532-000e-493a-a45a-02a4326beede" containerName="main" Jul 17 18:57:57.251232 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251235 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ee6532-000e-493a-a45a-02a4326beede" containerName="main" Jul 17 18:57:57.251405 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251283 2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f50599-cdd4-47fa-9a14-5e429da5424f" containerName="main" Jul 17 18:57:57.251405 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251292 2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f50599-cdd4-47fa-9a14-5e429da5424f" containerName="main" Jul 17 18:57:57.251501 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251454 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ee6532-000e-493a-a45a-02a4326beede" containerName="main" Jul 17 18:57:57.251501 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.251468 2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f50599-cdd4-47fa-9a14-5e429da5424f" containerName="main" Jul 17 18:57:57.255519 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.255496 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.258714 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.258693 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s9n5\"/\"kube-root-ca.crt\"" Jul 17 18:57:57.259246 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.259228 2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s9n5\"/\"openshift-service-ca.crt\"" Jul 17 18:57:57.259338 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.259253 2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5s9n5\"/\"default-dockercfg-dw4zq\"" Jul 17 18:57:57.263486 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.263205 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg"] Jul 17 18:57:57.375894 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.375859 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-podres\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.375894 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.375896 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-proc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.376097 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.375926 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-sys\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.376097 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.376023 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-lib-modules\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.376097 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.376057 2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrckc\" (UniqueName: \"kubernetes.io/projected/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-kube-api-access-vrckc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476528 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476495 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-lib-modules\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476682 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476536 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrckc\" (UniqueName: \"kubernetes.io/projected/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-kube-api-access-vrckc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476682 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476594 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-podres\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476682 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476623 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-proc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476682 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476650 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-lib-modules\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476682 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476672 2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-sys\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476694 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-proc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476771 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-sys\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.476875 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.476783 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-podres\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.488137 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.488108 2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrckc\" (UniqueName: \"kubernetes.io/projected/e8d7fb5f-6c20-4988-8068-eeffc3e8c937-kube-api-access-vrckc\") pod \"perf-node-gather-daemonset-bvfhg\" (UID: \"e8d7fb5f-6c20-4988-8068-eeffc3e8c937\") " pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.566776 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.566752 2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:57.693003 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:57.692956 2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg"] Jul 17 18:57:57.694965 ip-10-0-136-225 kubenswrapper[2587]: W0717 18:57:57.694934 2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8d7fb5f_6c20_4988_8068_eeffc3e8c937.slice/crio-245d84aeaef60f7a0ed7ba1a7c60b0324e57238529365dad319aa30ad367b795 WatchSource:0}: Error finding container 245d84aeaef60f7a0ed7ba1a7c60b0324e57238529365dad319aa30ad367b795: Status 404 returned error can't find the container with id 245d84aeaef60f7a0ed7ba1a7c60b0324e57238529365dad319aa30ad367b795 Jul 17 18:57:58.306896 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.306862 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" event={"ID":"e8d7fb5f-6c20-4988-8068-eeffc3e8c937","Type":"ContainerStarted","Data":"37ff75fbe89da6f7685987b5d88373853cc488ada7a92d1ab48ae9e5062a9477"} Jul 17 18:57:58.307305 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.306918 2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" event={"ID":"e8d7fb5f-6c20-4988-8068-eeffc3e8c937","Type":"ContainerStarted","Data":"245d84aeaef60f7a0ed7ba1a7c60b0324e57238529365dad319aa30ad367b795"} Jul 17 18:57:58.307305 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.306941 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:57:58.324050 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.324005 2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" podStartSLOduration=1.323975816 podStartE2EDuration="1.323975816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:58.32164466 +0000 UTC m=+2885.140415923" watchObservedRunningTime="2026-07-17 18:57:58.323975816 +0000 UTC m=+2885.142747079" Jul 17 18:57:58.694500 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.694426 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fbccc4567-xc8gm_60c1feac-96bb-475a-9c2f-d89b0bbdef99/console/0.log" Jul 17 18:57:58.729300 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:57:58.729277 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-jgppc_640f9ff7-4abf-4882-b8e6-1e9225680878/download-server/0.log" Jul 17 18:58:00.380026 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:00.379974 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8w8gb_dc9377ef-9af1-4c91-a65e-c831b6874962/dns/0.log" Jul 17 18:58:00.400825 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:00.400803 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8w8gb_dc9377ef-9af1-4c91-a65e-c831b6874962/kube-rbac-proxy/0.log" Jul 17 18:58:00.526150 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:00.526127 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-csmjl_e43c7842-165d-4505-b7a3-803c73c7c01d/dns-node-resolver/0.log" Jul 17 18:58:01.095691 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:01.095665 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f5587f5bc-ksm26_a91c3fa1-725a-4c89-8dfd-213791366ae5/registry/0.log" Jul 17 18:58:01.216623 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:01.216599 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lqzqv_28775f48-6fd5-429e-bf24-5567776b8861/node-ca/0.log" Jul 17 18:58:02.402845 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:02.402803 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-vwtnq_00f83132-f55f-42d9-91d0-e95d5e4af89e/kube-auth-proxy/0.log" Jul 17 18:58:03.389253 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:03.389224 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z46x2_bff42af5-1a78-43de-a2c6-2dabf7412bd7/serve-healthcheck-canary/0.log" Jul 17 18:58:03.947933 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:03.947907 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n57qg_87654b77-0f45-4df5-ba9e-d6002088b336/kube-rbac-proxy/0.log" Jul 17 18:58:03.971615 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:03.971591 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n57qg_87654b77-0f45-4df5-ba9e-d6002088b336/exporter/0.log" Jul 17 18:58:03.994567 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:03.994547 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n57qg_87654b77-0f45-4df5-ba9e-d6002088b336/extractor/0.log" Jul 17 18:58:04.322625 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:04.322594 2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5s9n5/perf-node-gather-daemonset-bvfhg" Jul 17 18:58:06.849933 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:06.849903 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-99g9h_7028b7bf-1c02-4809-8879-ef8088f88e68/manager/0.log" Jul 17 18:58:06.871977 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:06.871950 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9846c54f8-99g9h_7028b7bf-1c02-4809-8879-ef8088f88e68/copy-manifests/0.log" Jul 17 18:58:08.290418 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:08.290386 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cf59f64f5-nrkl2_7b5adc00-dc8b-4d16-bd02-0cfefa904820/manager/0.log" Jul 17 18:58:08.348357 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:08.348328 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-vwtnq_00f83132-f55f-42d9-91d0-e95d5e4af89e/kube-auth-proxy/0.log" Jul 17 18:58:09.863997 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:09.863962 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79874b6-vwtnq_00f83132-f55f-42d9-91d0-e95d5e4af89e/kube-auth-proxy/0.log" Jul 17 18:58:13.304799 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:13.304767 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-52rms_7f8943d1-361a-4dee-b7f4-5e97f511fb6a/migrator/0.log" Jul 17 18:58:13.327439 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:13.327418 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-52rms_7f8943d1-361a-4dee-b7f4-5e97f511fb6a/graceful-termination/0.log" Jul 17 18:58:14.668009 ip-10-0-136-225 kubenswrapper[2587]: I0717 18:58:14.667969 2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2n6k9_358e0c4e-84da-47c7-9c43-07c395d6d242/kube-multus/0.log"