Jul 10 20:35:42.875040 ip-10-0-142-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 20:35:42.875055 ip-10-0-142-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 20:35:42.875064 ip-10-0-142-194 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 20:35:42.875397 ip-10-0-142-194 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 20:35:53.007136 ip-10-0-142-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 20:35:53.007154 ip-10-0-142-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3704522599534353af46dad38a436103 -- Jul 10 20:37:49.263567 ip-10-0-142-194 systemd[1]: Starting Kubernetes Kubelet... Jul 10 20:37:49.714381 ip-10-0-142-194 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:37:49.714381 ip-10-0-142-194 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 20:37:49.714381 ip-10-0-142-194 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:37:49.714381 ip-10-0-142-194 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 20:37:49.714381 ip-10-0-142-194 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:37:49.715604 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.715417 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 20:37:49.720330 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720303 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.720330 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720327 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.720330 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720333 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720336 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720339 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720342 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720344 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720347 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720350 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720352 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720355 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720357 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720360 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720363 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720366 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720368 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720371 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720380 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720383 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720386 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720388 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.720439 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720391 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720393 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720396 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720398 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720401 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720403 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720406 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720408 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720411 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720413 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720415 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720418 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720420 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720422 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720425 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720429 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720431 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720434 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720436 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720439 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.720885 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720442 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720444 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720447 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720450 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720452 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720455 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720457 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720460 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720462 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720465 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720468 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720470 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720473 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720477 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720481 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720485 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720489 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720492 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720495 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.721386 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720498 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720500 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720503 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720505 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720508 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720511 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720513 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720517 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720520 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720523 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720525 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720528 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720531 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720535 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720538 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720543 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720546 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720549 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720552 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720554 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.721886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720557 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720559 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720562 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720568 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720571 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720573 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720576 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720578 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720580 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720583 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720586 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720590 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720592 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720595 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720597 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720600 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720603 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720605 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.720607 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721471 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.722376 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721481 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721484 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721487 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721490 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721493 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721496 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721499 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721503 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721507 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721510 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721513 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721516 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721518 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721521 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721524 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721527 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721536 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721539 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721542 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.722855 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721544 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721547 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721550 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721553 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721555 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721558 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721560 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721562 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721565 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721567 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721569 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721572 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721574 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721579 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721583 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721586 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721588 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721591 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721594 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721596 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.723341 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721598 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721601 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721604 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721607 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721609 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721612 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721615 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721618 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721620 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721622 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721641 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721645 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721647 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721650 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721653 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721656 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721658 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721662 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721665 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721668 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.723829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721672 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721675 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721678 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721681 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721684 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721687 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721689 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721691 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721694 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721696 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721699 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721701 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721705 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721708 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721711 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721713 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721716 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721718 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721721 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721724 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.724344 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721726 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721728 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721731 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721739 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721742 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721745 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721748 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721750 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721753 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721755 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721757 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721760 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721763 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721765 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721768 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721770 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721773 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721775 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.721778 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722258 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.724828 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722264 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722267 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722270 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722273 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722276 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722279 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722282 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722285 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722287 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722290 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722292 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722295 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722298 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722300 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722303 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722306 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722309 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722311 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722314 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722316 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.725339 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722319 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722321 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722323 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722326 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722328 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722333 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722337 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722339 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722342 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722345 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722347 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722350 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722352 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722354 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722357 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722359 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722362 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722367 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722369 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.725832 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722372 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722374 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722377 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722379 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722382 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722385 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722388 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722390 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722393 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722396 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722399 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722401 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722405 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722408 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722411 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722414 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722417 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722419 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722423 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.726310 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722426 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722429 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722431 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722434 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722437 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722440 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722443 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722446 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722448 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722451 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722453 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722456 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722459 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722461 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722463 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722466 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722470 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722473 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722475 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.726825 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722477 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722480 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722482 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722485 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722487 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722491 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722494 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722496 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722499 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722501 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722504 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722506 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722509 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722512 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722514 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722517 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722519 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722521 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722524 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722526 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.727286 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.722528 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722607 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722615 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722621 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722626 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722630 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722634 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722638 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722643 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722646 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722649 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722653 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722656 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722659 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722662 2585 flags.go:64] FLAG: --cgroup-root="" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722665 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722668 2585 flags.go:64] FLAG: --client-ca-file="" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722671 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722675 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722679 2585 flags.go:64] FLAG: --cluster-domain="" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722682 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722685 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722688 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722692 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 20:37:49.727788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722696 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722699 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722702 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722705 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722709 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722712 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722715 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722718 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722721 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722725 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722728 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722731 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722734 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722737 2585 flags.go:64] FLAG: --enable-server="true" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722740 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722745 2585 flags.go:64] FLAG: --event-burst="100" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722748 2585 flags.go:64] FLAG: --event-qps="50" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722751 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722754 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722757 2585 flags.go:64] FLAG: --eviction-hard="" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722760 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722763 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722766 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722769 2585 flags.go:64] FLAG: --eviction-soft="" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722772 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 20:37:49.728385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722776 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722780 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722783 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722786 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722789 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722791 2585 flags.go:64] FLAG: --feature-gates="" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722796 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722799 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722802 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722806 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722808 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722811 2585 flags.go:64] FLAG: --help="false" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722815 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-142-194.ec2.internal" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722818 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722821 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722823 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722826 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722830 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722833 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722836 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722839 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722841 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722844 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722847 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 20:37:49.729091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722850 2585 flags.go:64] FLAG: --kube-reserved="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722853 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722856 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722859 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722863 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722865 2585 flags.go:64] FLAG: --lock-file="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722868 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722871 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722874 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722879 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722883 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722885 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722888 2585 flags.go:64] FLAG: --logging-format="text" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722891 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722895 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722897 2585 flags.go:64] FLAG: --manifest-url="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722900 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722904 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722909 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722914 2585 flags.go:64] FLAG: --max-pods="110" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722917 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722920 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722922 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722925 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722928 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 20:37:49.729698 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722931 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722934 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722942 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722945 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722948 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722951 2585 flags.go:64] FLAG: --pod-cidr="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722954 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722959 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722962 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722969 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722972 2585 flags.go:64] FLAG: --port="10250" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722974 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722977 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06f5b081b43db7dd7" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722980 2585 flags.go:64] FLAG: --qos-reserved="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722984 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722986 2585 flags.go:64] FLAG: --register-node="true" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722989 2585 flags.go:64] FLAG: --register-with-taints="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722993 2585 flags.go:64] FLAG: --registry-burst="10" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722996 2585 flags.go:64] FLAG: --registry-qps="5" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.722999 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723002 2585 flags.go:64] FLAG: --reserved-memory="" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723008 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723011 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723014 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723016 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 20:37:49.730353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723021 2585 flags.go:64] FLAG: --runonce="false" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723024 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723027 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723030 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723033 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723036 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723039 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723042 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723045 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723048 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723051 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723054 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723057 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723060 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723063 2585 flags.go:64] FLAG: --system-cgroups="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723065 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723071 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723074 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723077 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723081 2585 flags.go:64] FLAG: --tls-min-version="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723084 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723087 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723090 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723093 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723096 2585 flags.go:64] FLAG: --v="2" Jul 10 20:37:49.730986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723100 2585 flags.go:64] FLAG: --version="false" Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723105 2585 flags.go:64] FLAG: --vmodule="" Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723109 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.723116 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723257 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723263 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723267 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723273 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723276 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723279 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723281 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723284 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723286 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723289 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723292 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723294 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723297 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723299 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723302 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723305 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.731984 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723307 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723309 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723312 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723314 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723317 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723319 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723322 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723324 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723327 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723329 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723332 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723334 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723337 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723340 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723343 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723348 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723351 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723354 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723357 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723362 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.732702 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723366 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723368 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723371 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723374 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723376 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723379 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723381 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723384 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723388 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723392 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723395 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723397 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723400 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723403 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723405 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723408 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723410 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723413 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723415 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.733184 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723418 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723420 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723422 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723425 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723427 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723430 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723432 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723435 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723439 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723442 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723444 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723447 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723451 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723453 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723456 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723458 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723461 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723463 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723465 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723468 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.733794 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723471 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723473 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723476 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723478 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723481 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723483 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723486 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723488 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723491 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723494 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723498 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723500 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723504 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723506 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723509 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723511 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723514 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723517 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723519 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723522 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.734295 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723525 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723529 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723531 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.723534 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.724297 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.733214 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.733259 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.733282 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.733288 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733351 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733356 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733359 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733363 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733365 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.734783 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733368 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733371 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733374 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733378 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733382 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733385 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733387 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733390 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733392 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733395 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733400 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733404 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733407 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733410 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733412 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733415 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733418 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733420 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733423 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.735122 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733427 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733430 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733433 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733436 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733439 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733442 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733445 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733448 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733450 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733453 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733456 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733458 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733460 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733463 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733465 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733468 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733470 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733473 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733475 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.735604 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733478 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733480 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733483 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733487 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733489 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733492 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733494 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733496 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733500 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733504 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733506 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733509 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733512 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733515 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733518 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733521 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733524 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733527 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733529 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.736070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733532 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733535 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733538 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733540 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733543 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733545 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733547 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733550 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733552 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733555 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733557 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733560 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733563 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733565 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733568 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733571 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733574 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733577 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733580 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733582 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.736549 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733585 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733587 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733589 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733592 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733595 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733597 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733600 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733603 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733605 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733608 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733610 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733613 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733616 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733618 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733620 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733623 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.737075 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733625 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.733639 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733768 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733774 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733777 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733780 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733783 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733785 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733788 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733790 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733793 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733796 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733798 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733801 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733804 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:37:49.737580 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733807 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733810 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733813 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733816 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733818 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733821 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733823 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733826 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733828 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733831 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733833 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733837 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733839 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733842 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733844 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733847 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733849 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733852 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733854 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:37:49.737988 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733857 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733859 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733862 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733864 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733867 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733869 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733871 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733874 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733877 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733879 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733882 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733885 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733887 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733891 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733893 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733896 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733898 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733901 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733904 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733906 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:37:49.738544 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733908 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733911 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733913 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733916 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733918 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733920 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733923 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733925 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733928 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733931 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733933 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733937 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733940 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733943 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733946 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733948 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733952 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733955 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733958 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:37:49.739066 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733961 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733964 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733966 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733968 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733971 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733974 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733978 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733981 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733984 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733986 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733989 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733992 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733994 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.733997 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734000 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734002 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734004 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734007 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734009 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:37:49.739608 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734012 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734015 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734017 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734019 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734022 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734024 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734026 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734029 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:49.734031 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.734036 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.734168 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.736506 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.738319 2585 server.go:1019] "Starting client certificate rotation" Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.738449 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:37:49.740124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.738507 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:37:49.741052 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.741030 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:37:49.742425 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.742398 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:37:49.759195 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.759166 2585 log.go:25] "Validated CRI v1 runtime API" Jul 10 20:37:49.761641 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.761619 2585 log.go:25] "Validated CRI v1 image API" Jul 10 20:37:49.762533 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.762511 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 20:37:49.765198 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.765172 2585 fs.go:135] Filesystem UUIDs: map[46e9d1cc-9b6d-4ca4-9982-3efa91dc4556:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e5d8751c-18de-4919-9715-94f8d7d9e942:/dev/nvme0n1p3] Jul 10 20:37:49.765299 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.765195 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 20:37:49.768103 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.768083 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:37:49.770769 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.770633 2585 manager.go:216] Machine: {Timestamp:2026-07-10 20:37:49.769282137 +0000 UTC m=+0.376038730 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104262 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24ede9615d2690e0779dd818045bc2 SystemUUID:ec24ede9-615d-2690-e077-9dd818045bc2 BootID:37045225-9953-4353-af46-dad38a436103 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c3:42:d6:3e:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c3:42:d6:3e:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:f5:37:14:2f:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 20:37:49.770769 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.770760 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 20:37:49.770906 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.770855 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 20:37:49.775883 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.775672 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 20:37:49.776028 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.775881 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 20:37:49.776799 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.776787 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 20:37:49.776848 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.776801 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 20:37:49.776848 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.776809 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:37:49.777769 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.777757 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:37:49.779350 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.779336 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 20:37:49.782681 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782667 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:37:49.782819 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782809 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 20:37:49.782856 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782840 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 10 20:37:49.782856 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782852 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 20:37:49.782912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782867 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 20:37:49.782912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782877 2585 kubelet.go:398] "Adding apiserver pod source" Jul 10 20:37:49.782912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.782886 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 20:37:49.783720 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.783706 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:37:49.783785 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.783728 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:37:49.786775 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.786757 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 20:37:49.788898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.788885 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 20:37:49.788951 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.788916 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 20:37:49.789066 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789057 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 20:37:49.789096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789072 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 20:37:49.789096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789079 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 20:37:49.789096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789084 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 20:37:49.789096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789091 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 20:37:49.789096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.789097 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 20:37:49.790385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790373 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 20:37:49.790419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790386 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 20:37:49.790419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790393 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 20:37:49.790419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790399 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 20:37:49.790419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790408 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 20:37:49.790419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.790417 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 20:37:49.792537 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.792518 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-llkvm" Jul 10 20:37:49.792763 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.792749 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 20:37:49.792813 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.792766 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 20:37:49.793044 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.793010 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 20:37:49.793044 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.793024 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 20:37:49.796089 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.796075 2585 server.go:1268] "Started kubelet" Jul 10 20:37:49.796195 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.796168 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 20:37:49.796269 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.796161 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 20:37:49.796317 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.796271 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 20:37:49.797151 ip-10-0-142-194 systemd[1]: Started Kubernetes Kubelet. Jul 10 20:37:49.798535 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.798518 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 20:37:49.798631 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.798619 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 10 20:37:49.799192 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.799175 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-llkvm" Jul 10 20:37:49.800075 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.800053 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 20:37:49.801346 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.801326 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 20:37:49.804586 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.804570 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 20:37:49.804818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.804708 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 20:37:49.804818 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.804738 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:49.805381 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805353 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 20:37:49.805462 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805391 2585 factory.go:55] Registering systemd factory Jul 10 20:37:49.805462 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805407 2585 factory.go:223] Registration of the systemd container factory successfully Jul 10 20:37:49.805462 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805462 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 20:37:49.805579 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805472 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 20:37:49.805579 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805540 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 10 20:37:49.805579 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805551 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 10 20:37:49.805694 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805649 2585 factory.go:153] Registering CRI-O factory Jul 10 20:37:49.805694 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805664 2585 factory.go:223] Registration of the crio container factory successfully Jul 10 20:37:49.805773 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805721 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 20:37:49.805773 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805748 2585 factory.go:103] Registering Raw factory Jul 10 20:37:49.805773 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.805765 2585 manager.go:1196] Started watching for new ooms in manager Jul 10 20:37:49.806187 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.806174 2585 manager.go:349] Starting recovery of all containers Jul 10 20:37:49.811543 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.811407 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:49.815156 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.815131 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-194.ec2.internal\" not found" node="ip-10-0-142-194.ec2.internal" Jul 10 20:37:49.822653 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.822459 2585 manager.go:354] Recovery completed Jul 10 20:37:49.826941 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.826925 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:49.828876 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.828856 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:49.828970 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.828893 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:49.828970 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.828910 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:49.829488 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.829475 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 20:37:49.829551 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.829488 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:49.829551 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.829487 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 20:37:49.829551 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.829541 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:37:49.830175 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.830160 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:49.830223 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.830187 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:49.830223 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.830197 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:49.831020 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.831009 2585 policy_none.go:49] "None policy: Start" Jul 10 20:37:49.831051 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.831025 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 20:37:49.831051 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.831036 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 20:37:49.832419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.832408 2585 policy_none.go:47] "Start" Jul 10 20:37:49.869916 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.869894 2585 manager.go:337] "Starting Device Plugin manager" Jul 10 20:37:49.870031 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.869943 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 20:37:49.870031 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.869957 2585 server.go:87] "Starting device plugin registration server" Jul 10 20:37:49.870314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.870300 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 20:37:49.870380 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.870316 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 20:37:49.870441 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.870425 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 20:37:49.870513 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.870501 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 20:37:49.870553 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.870515 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 20:37:49.871074 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.871057 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 20:37:49.871164 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.871105 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:49.970536 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.970431 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:49.971730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.971492 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:49.971730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.971525 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:49.971730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.971535 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:49.971730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.971569 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-194.ec2.internal" Jul 10 20:37:49.977258 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.977214 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 20:37:49.978731 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.978697 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 20:37:49.978863 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.978742 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 20:37:49.978863 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.978771 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 20:37:49.978863 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.978810 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 20:37:49.980593 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.980574 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-194.ec2.internal" Jul 10 20:37:49.980884 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:49.980597 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-194.ec2.internal\": node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:49.981419 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:49.981403 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:50.008268 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.008221 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.079351 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.079320 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal"] Jul 10 20:37:50.079439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.079404 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:50.080539 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.080523 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:50.080618 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.080553 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:50.080618 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.080564 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:50.082363 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.082351 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:50.082483 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.082467 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.082540 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.082504 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:50.083339 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083314 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:50.083450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083347 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:50.083450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083356 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:50.083450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083314 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:50.083450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083432 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:50.083450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.083448 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:50.084655 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.084642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.084697 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.084667 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:37:50.085388 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.085363 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:37:50.085388 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.085389 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:37:50.085519 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.085401 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:37:50.100845 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.100821 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-194.ec2.internal\" not found" node="ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.104764 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.104746 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-194.ec2.internal\" not found" node="ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.107413 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.107396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/197bfbeb65b7318ec9e668e91130f11b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"197bfbeb65b7318ec9e668e91130f11b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.107492 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.107426 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.107492 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.107448 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.108947 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.108934 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.208639 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/197bfbeb65b7318ec9e668e91130f11b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"197bfbeb65b7318ec9e668e91130f11b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.208807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208645 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.208807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208684 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.208807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208715 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/197bfbeb65b7318ec9e668e91130f11b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-194.ec2.internal\" (UID: \"197bfbeb65b7318ec9e668e91130f11b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.208807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208729 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.208807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.208741 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc7c1f32995e2f09dba311721291d26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal\" (UID: \"ebc7c1f32995e2f09dba311721291d26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.209456 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.209435 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.310599 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.310527 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.405826 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.405780 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.408161 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.408145 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:50.411202 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.411185 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.511677 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.511632 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.612647 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.612554 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.713131 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.713101 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.738142 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.738114 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 20:37:50.738653 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.738358 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 20:37:50.738653 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.738358 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 20:37:50.805281 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.805201 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 20:32:50 +0000 UTC" deadline="2028-04-22 22:47:29.925065936 +0000 UTC" Jul 10 20:37:50.805281 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:50.805277 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15650h9m39.119793851s" Jul 10 20:37:50.813412 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.813394 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:50.913736 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:50.913638 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:51.013837 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.013800 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-194.ec2.internal\" not found" Jul 10 20:37:51.112039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.112013 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:51.138113 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.138087 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:51.138113 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:51.138092 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc7c1f32995e2f09dba311721291d26.slice/crio-ffb2a7ecf9979f9251020b7cf20aca4e90108e7dfdcf163fda4cf7225c5f1da0 WatchSource:0}: Error finding container ffb2a7ecf9979f9251020b7cf20aca4e90108e7dfdcf163fda4cf7225c5f1da0: Status 404 returned error can't find the container with id ffb2a7ecf9979f9251020b7cf20aca4e90108e7dfdcf163fda4cf7225c5f1da0 Jul 10 20:37:51.138598 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:51.138578 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197bfbeb65b7318ec9e668e91130f11b.slice/crio-5084d79147538a48f1e751805ee8572559ee74f318232aa7b5d4ebbfb7f02f38 WatchSource:0}: Error finding container 5084d79147538a48f1e751805ee8572559ee74f318232aa7b5d4ebbfb7f02f38: Status 404 returned error can't find the container with id 5084d79147538a48f1e751805ee8572559ee74f318232aa7b5d4ebbfb7f02f38 Jul 10 20:37:51.142267 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.142253 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:37:51.205467 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.205364 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" Jul 10 20:37:51.216672 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.216641 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:37:51.217923 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.217906 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" Jul 10 20:37:51.225676 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.225659 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:37:51.776810 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.776765 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:51.785677 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.785649 2585 apiserver.go:52] "Watching apiserver" Jul 10 20:37:51.791253 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.791219 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 20:37:51.791810 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.791761 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-blr66","openshift-dns/node-resolver-9zzgw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal","openshift-multus/multus-additional-cni-plugins-dl78j","openshift-network-operator/iptables-alerter-szhnj","kube-system/global-pull-secret-syncer-xdr8l","kube-system/konnectivity-agent-rmqm6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz","openshift-image-registry/node-ca-v6xrm","openshift-multus/multus-dgdgm","openshift-multus/network-metrics-daemon-vnrsl","openshift-network-diagnostics/network-check-target-zr2tc","openshift-ovn-kubernetes/ovnkube-node-cxw5l","kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal"] Jul 10 20:37:51.795003 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.794979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.796126 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.795934 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.796924 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.796898 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xbdqz\"" Jul 10 20:37:51.797064 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.797045 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 20:37:51.797137 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.796898 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 20:37:51.797550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.797530 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.799737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.798452 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.799737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.798662 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 20:37:51.799737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.798856 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2hxl\"" Jul 10 20:37:51.799737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.799078 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 20:37:51.799737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.799526 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.802378 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802357 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 20:37:51.802378 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802372 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.802650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802566 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.802650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802635 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 20:37:51.802776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802650 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 20:37:51.802832 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802813 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87bjh\"" Jul 10 20:37:51.802948 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802878 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.803010 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.802985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.803928 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.803910 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.804022 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.803983 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:37:51.804669 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.804468 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.804966 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.804948 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 20:37:51.805072 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.804979 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 20:37:51.805250 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805233 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4f75q\"" Jul 10 20:37:51.805341 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805244 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 20:37:51.805530 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805514 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 20:37:51.805619 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805577 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.805791 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805722 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sgwbj\"" Jul 10 20:37:51.806010 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805790 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.806010 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.805519 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.806133 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.806064 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.806883 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.806863 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 20:37:51.808097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.807646 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.808097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.807691 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 20:37:51.808097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.807822 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbns7\"" Jul 10 20:37:51.808097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.807944 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.808097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.808067 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.808972 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.808918 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:51.812812 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.812481 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.815336 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.813115 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-975lm\"" Jul 10 20:37:51.815336 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.813833 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.815336 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.814487 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.815535 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.815360 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 20:37:51.815755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.815733 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 20:37:51.815869 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.815838 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d4hb4\"" Jul 10 20:37:51.815990 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.815912 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 20:37:51.816099 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.816082 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4zph8\"" Jul 10 20:37:51.816860 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.816502 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:51.816860 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.816631 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:37:51.817030 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.816866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-systemd\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817030 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.816912 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-tuned\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817030 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.816947 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-slash\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817030 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817014 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-config\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gff6p\" (UniqueName: \"kubernetes.io/projected/83624a0b-c4ce-4308-bcb4-bcae87df2139-kube-api-access-gff6p\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817058 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a989078a-a850-4db3-9042-d195a3e461e7-agent-certs\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.817249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817088 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:37:51.817249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-ovn\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817269 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-sys\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817304 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817363 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-host\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-bin\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817450 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-env-overrides\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjf7\" (UniqueName: \"kubernetes.io/projected/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-kube-api-access-zdjf7\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6g2b\" (UniqueName: \"kubernetes.io/projected/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-kube-api-access-j6g2b\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-var-lib-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-host-slash\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-kubernetes\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.817827 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-sys-fs\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.818185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-var-lib-kubelet\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.818185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817869 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-conf\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.818185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817895 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-system-cni-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.818185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.817924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-device-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.818812 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818467 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwwh\" (UniqueName: \"kubernetes.io/projected/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-kube-api-access-7lwwh\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.818901 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb239305-035b-4c95-bda0-aabdb6f65431-host\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.818901 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818875 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.819019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818903 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzxm\" (UniqueName: \"kubernetes.io/projected/88918d10-2931-4413-9080-2f6140a21cda-kube-api-access-mdzxm\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.819019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.819019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.818981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.819019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cnibin\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.819283 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.819283 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.819283 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-systemd-units\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819283 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819229 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-netd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819283 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-registration-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819372 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-run\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-tmp\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-netns\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-etc-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.819572 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819615 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysconfig\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819640 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-systemd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819691 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-kubelet-config\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-dbus\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819756 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-iptables-alerter-script\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.819805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-lib-modules\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-log-socket\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819902 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql4d\" (UniqueName: \"kubernetes.io/projected/cb239305-035b-4c95-bda0-aabdb6f65431-kube-api-access-fql4d\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-socket-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.819997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-modprobe-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820024 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a989078a-a850-4db3-9042-d195a3e461e7-konnectivity-ca\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-script-lib\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-kubelet\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-node-log\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820171 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.820209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820195 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-os-release\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.820733 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820250 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb239305-035b-4c95-bda0-aabdb6f65431-serviceca\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.820733 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.820277 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.844082 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.844049 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7z7c4" Jul 10 20:37:51.851632 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.851606 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7z7c4" Jul 10 20:37:51.887256 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.887226 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:37:51.907104 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.907074 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 20:37:51.920644 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql4d\" (UniqueName: \"kubernetes.io/projected/cb239305-035b-4c95-bda0-aabdb6f65431-kube-api-access-fql4d\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.920830 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-socket-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.920830 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920807 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-socket-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.920912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-cnibin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.920912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-hostroot\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.921000 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920910 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-modprobe-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.921000 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a989078a-a850-4db3-9042-d195a3e461e7-konnectivity-ca\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.921000 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.920981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-script-lib\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921015 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74s4s\" (UniqueName: \"kubernetes.io/projected/f525f08f-e462-48e9-8530-85773208a545-kube-api-access-74s4s\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:51.921122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-kubelet\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-node-log\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921126 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-kubelet\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-modprobe-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.921254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921230 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921189 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-node-log\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-os-release\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921306 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-os-release\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb239305-035b-4c95-bda0-aabdb6f65431-serviceca\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-kubelet\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-systemd\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-tuned\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-slash\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-config\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.921532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921509 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-os-release\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-systemd\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a989078a-a850-4db3-9042-d195a3e461e7-konnectivity-ca\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-slash\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnjh\" (UniqueName: \"kubernetes.io/projected/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-kube-api-access-rlnjh\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gff6p\" (UniqueName: \"kubernetes.io/projected/83624a0b-c4ce-4308-bcb4-bcae87df2139-kube-api-access-gff6p\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a989078a-a850-4db3-9042-d195a3e461e7-agent-certs\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921738 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-ovn\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-sys\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921806 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb239305-035b-4c95-bda0-aabdb6f65431-serviceca\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921834 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-host\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921856 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-bin\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921883 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921872 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-env-overrides\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.921940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjf7\" (UniqueName: \"kubernetes.io/projected/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-kube-api-access-zdjf7\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.921964 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922013 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6g2b\" (UniqueName: \"kubernetes.io/projected/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-kube-api-access-j6g2b\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-script-lib\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922045 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-var-lib-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922114 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-ovn\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-host-slash\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922149 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-sys\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-bin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-var-lib-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922285 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-host-slash\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:51.922369 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:52.42232692 +0000 UTC m=+3.029083498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-env-overrides\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-daemon-config\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.922818 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k29t\" (UniqueName: \"kubernetes.io/projected/56ff23da-1f36-4fc2-b33e-230ba5252990-kube-api-access-7k29t\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-hosts-file\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-kubernetes\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-sys-fs\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-multus-certs\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-var-lib-kubelet\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-kubernetes\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-conf\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-bin\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-system-cni-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922676 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-var-lib-kubelet\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-host\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922728 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-sys-fs\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-device-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922782 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-system-cni-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-conf\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922813 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.923580 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922994 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923012 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-etc-kubernetes\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.922846 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-device-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwwh\" (UniqueName: \"kubernetes.io/projected/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-kube-api-access-7lwwh\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923094 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb239305-035b-4c95-bda0-aabdb6f65431-host\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923165 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzxm\" (UniqueName: \"kubernetes.io/projected/88918d10-2931-4413-9080-2f6140a21cda-kube-api-access-mdzxm\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb239305-035b-4c95-bda0-aabdb6f65431-host\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-k8s-cni-cncf-io\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923214 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-system-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-netns\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923388 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cnibin\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.924428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovnkube-config\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923450 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysctl-d\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923499 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cnibin\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923545 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-multus\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-conf-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-systemd-units\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-netd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923747 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-cni-netd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923775 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-systemd-units\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-registration-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923921 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923930 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-run\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.923995 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88918d10-2931-4413-9080-2f6140a21cda-registration-dir\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-tmp\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925208 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924077 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-run\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924094 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-netns\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-etc-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-host-run-netns\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-cni-binary-copy\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924272 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-etc-openvswitch\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-socket-dir-parent\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924342 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-tmp-dir\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysconfig\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-systemd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-kubelet-config\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-dbus\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924471 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-sysconfig\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-iptables-alerter-script\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-lib-modules\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924525 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-kubelet-config\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.925898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924541 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-log-socket\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-run-systemd\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-log-socket\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924712 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83624a0b-c4ce-4308-bcb4-bcae87df2139-lib-modules\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.924802 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1842cdd3-d259-4106-9d65-618540a99480-dbus\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.925011 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.925360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-iptables-alerter-script\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.925606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-etc-tuned\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.926566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.926780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.926596 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a989078a-a850-4db3-9042-d195a3e461e7-agent-certs\") pod \"konnectivity-agent-rmqm6\" (UID: \"a989078a-a850-4db3-9042-d195a3e461e7\") " pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:51.927735 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.927679 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83624a0b-c4ce-4308-bcb4-bcae87df2139-tmp\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.935387 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.934947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjf7\" (UniqueName: \"kubernetes.io/projected/8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c-kube-api-access-zdjf7\") pod \"ovnkube-node-cxw5l\" (UID: \"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:51.935387 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.935092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzxm\" (UniqueName: \"kubernetes.io/projected/88918d10-2931-4413-9080-2f6140a21cda-kube-api-access-mdzxm\") pod \"aws-ebs-csi-driver-node-l6jdz\" (UID: \"88918d10-2931-4413-9080-2f6140a21cda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:51.938540 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.938516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwwh\" (UniqueName: \"kubernetes.io/projected/a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0-kube-api-access-7lwwh\") pod \"multus-additional-cni-plugins-dl78j\" (UID: \"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0\") " pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:51.938831 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.938812 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gff6p\" (UniqueName: \"kubernetes.io/projected/83624a0b-c4ce-4308-bcb4-bcae87df2139-kube-api-access-gff6p\") pod \"tuned-blr66\" (UID: \"83624a0b-c4ce-4308-bcb4-bcae87df2139\") " pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:51.938991 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.938972 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql4d\" (UniqueName: \"kubernetes.io/projected/cb239305-035b-4c95-bda0-aabdb6f65431-kube-api-access-fql4d\") pod \"node-ca-v6xrm\" (UID: \"cb239305-035b-4c95-bda0-aabdb6f65431\") " pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:51.940680 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.940653 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6g2b\" (UniqueName: \"kubernetes.io/projected/780f7dc2-8673-4890-bab2-2e0f6ef42c3e-kube-api-access-j6g2b\") pod \"iptables-alerter-szhnj\" (UID: \"780f7dc2-8673-4890-bab2-2e0f6ef42c3e\") " pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:51.985689 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.985629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"197bfbeb65b7318ec9e668e91130f11b","Type":"ContainerStarted","Data":"5084d79147538a48f1e751805ee8572559ee74f318232aa7b5d4ebbfb7f02f38"} Jul 10 20:37:51.987678 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:51.987640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"ebc7c1f32995e2f09dba311721291d26","Type":"ContainerStarted","Data":"ffb2a7ecf9979f9251020b7cf20aca4e90108e7dfdcf163fda4cf7225c5f1da0"} Jul 10 20:37:52.025206 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-cni-binary-copy\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025206 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025218 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-socket-dir-parent\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025470 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-tmp-dir\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.025673 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025648 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-tmp-dir\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.025772 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025740 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-socket-dir-parent\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025835 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025775 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-cnibin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025835 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025803 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-hostroot\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025943 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025834 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74s4s\" (UniqueName: \"kubernetes.io/projected/f525f08f-e462-48e9-8530-85773208a545-kube-api-access-74s4s\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:52.025943 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025864 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-kubelet\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025943 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025891 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-os-release\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025943 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025901 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-cni-binary-copy\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.025943 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025919 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnjh\" (UniqueName: \"kubernetes.io/projected/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-kube-api-access-rlnjh\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.025999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-bin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-daemon-config\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k29t\" (UniqueName: \"kubernetes.io/projected/56ff23da-1f36-4fc2-b33e-230ba5252990-kube-api-access-7k29t\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026094 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-hosts-file\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026120 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-multus-certs\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-etc-kubernetes\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:52.026218 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026204 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-cnibin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026227 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-k8s-cni-cncf-io\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-hostroot\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-system-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-netns\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-multus\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026394 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-conf-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-kubelet\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-os-release\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-conf-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-etc-kubernetes\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026527 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-bin\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.026572 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.026632 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:52.526613178 +0000 UTC m=+3.133369759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:52.026702 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026678 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-k8s-cni-cncf-io\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-daemon-config\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-system-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-hosts-file\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.026997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-multus-certs\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.027034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-multus-cni-dir\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.027058 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-var-lib-cni-multus\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.027515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.027082 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56ff23da-1f36-4fc2-b33e-230ba5252990-host-run-netns\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.033393 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.033366 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:37:52.033517 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.033399 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:37:52.033517 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.033415 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:52.033517 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.033500 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:52.533479374 +0000 UTC m=+3.140235955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:52.036467 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.036437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnjh\" (UniqueName: \"kubernetes.io/projected/cfa3b044-cb6f-41f8-b5d8-96fe2948004c-kube-api-access-rlnjh\") pod \"node-resolver-9zzgw\" (UID: \"cfa3b044-cb6f-41f8-b5d8-96fe2948004c\") " pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.036785 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.036764 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74s4s\" (UniqueName: \"kubernetes.io/projected/f525f08f-e462-48e9-8530-85773208a545-kube-api-access-74s4s\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:52.036868 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.036839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k29t\" (UniqueName: \"kubernetes.io/projected/56ff23da-1f36-4fc2-b33e-230ba5252990-kube-api-access-7k29t\") pod \"multus-dgdgm\" (UID: \"56ff23da-1f36-4fc2-b33e-230ba5252990\") " pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.117005 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.116964 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:37:52.125510 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.125483 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:37:52.138099 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.138072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dgdgm" Jul 10 20:37:52.143320 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.143283 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" Jul 10 20:37:52.153321 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.153284 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dl78j" Jul 10 20:37:52.163216 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.163193 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szhnj" Jul 10 20:37:52.172280 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.172249 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6xrm" Jul 10 20:37:52.181615 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.181584 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blr66" Jul 10 20:37:52.187891 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.187864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zzgw" Jul 10 20:37:52.430452 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.430359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:52.430627 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.430541 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:52.430691 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.430620 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:53.430597529 +0000 UTC m=+4.037354108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:52.531050 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.531010 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:52.531212 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.531164 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:52.531282 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.531234 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:53.531215374 +0000 UTC m=+4.137971935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:52.632005 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.631966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:52.632193 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.632122 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:37:52.632193 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.632146 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:37:52.632193 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.632161 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:52.632394 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.632234 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:53.632216036 +0000 UTC m=+4.238972620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:52.854049 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.853950 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:32:52 +0000 UTC" deadline="2028-01-06 15:43:23.600913928 +0000 UTC" Jul 10 20:37:52.854049 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.853988 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13075h5m30.746930232s" Jul 10 20:37:52.979562 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:52.979516 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:52.979768 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:52.979664 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:37:53.140217 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:53.138737 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780f7dc2_8673_4890_bab2_2e0f6ef42c3e.slice/crio-4e77b5de223ebb116ecfa9e58a5ada7cac201ed7bb9c50389f0ea0e46e814fa2 WatchSource:0}: Error finding container 4e77b5de223ebb116ecfa9e58a5ada7cac201ed7bb9c50389f0ea0e46e814fa2: Status 404 returned error can't find the container with id 4e77b5de223ebb116ecfa9e58a5ada7cac201ed7bb9c50389f0ea0e46e814fa2 Jul 10 20:37:53.162994 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:53.162962 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c85bfd1_981f_4a23_9eb8_70e5f3c57a1c.slice/crio-aab2087575a5b19ff9008afc3f07598f8dfa4e8ab1c7b740fd1359db60a8c566 WatchSource:0}: Error finding container aab2087575a5b19ff9008afc3f07598f8dfa4e8ab1c7b740fd1359db60a8c566: Status 404 returned error can't find the container with id aab2087575a5b19ff9008afc3f07598f8dfa4e8ab1c7b740fd1359db60a8c566 Jul 10 20:37:53.163228 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:53.163204 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ff23da_1f36_4fc2_b33e_230ba5252990.slice/crio-07ee687616580f4e2586229c2efe986973963c9272f795c0a76333d488bccecb WatchSource:0}: Error finding container 07ee687616580f4e2586229c2efe986973963c9272f795c0a76333d488bccecb: Status 404 returned error can't find the container with id 07ee687616580f4e2586229c2efe986973963c9272f795c0a76333d488bccecb Jul 10 20:37:53.164986 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:53.164960 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda989078a_a850_4db3_9042_d195a3e461e7.slice/crio-5c4a815b5df3f8d83c7ce468178ac5a8b7c6058171fddb6ff614cc14720cf74b WatchSource:0}: Error finding container 5c4a815b5df3f8d83c7ce468178ac5a8b7c6058171fddb6ff614cc14720cf74b: Status 404 returned error can't find the container with id 5c4a815b5df3f8d83c7ce468178ac5a8b7c6058171fddb6ff614cc14720cf74b Jul 10 20:37:53.166128 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:37:53.166103 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa3b044_cb6f_41f8_b5d8_96fe2948004c.slice/crio-838ad1669865098fe3db021c2d003277362718202d2033d8d20df58b96348cca WatchSource:0}: Error finding container 838ad1669865098fe3db021c2d003277362718202d2033d8d20df58b96348cca: Status 404 returned error can't find the container with id 838ad1669865098fe3db021c2d003277362718202d2033d8d20df58b96348cca Jul 10 20:37:53.439478 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.439275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:53.439646 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.439433 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:53.439646 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.439600 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:55.439583131 +0000 UTC m=+6.046339706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:53.540374 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.540340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:53.540541 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.540494 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:53.540584 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.540564 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:55.540546403 +0000 UTC m=+6.147302965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:53.641297 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.641163 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:53.641598 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.641410 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:37:53.641598 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.641433 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:37:53.641598 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.641447 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:53.641598 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.641516 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:55.641493394 +0000 UTC m=+6.248249962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:53.854543 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.854471 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:32:52 +0000 UTC" deadline="2027-12-25 03:08:18.196289536 +0000 UTC" Jul 10 20:37:53.854543 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.854542 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12774h30m24.341753057s" Jul 10 20:37:53.980803 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.980158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:53.980803 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.980321 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:37:53.981043 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:53.980864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:53.981043 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:53.980970 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:37:54.007607 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.007549 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" event={"ID":"197bfbeb65b7318ec9e668e91130f11b","Type":"ContainerStarted","Data":"363c954b1956d6c23127a381bad6200e1d05828852fae30c1cab34a6380d7cc8"} Jul 10 20:37:54.030935 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.030893 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6xrm" event={"ID":"cb239305-035b-4c95-bda0-aabdb6f65431","Type":"ContainerStarted","Data":"10c39a947ffe0428ae3e052810331254bff14570f84447ec3ec9f2d3183424a9"} Jul 10 20:37:54.042730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.042689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blr66" event={"ID":"83624a0b-c4ce-4308-bcb4-bcae87df2139","Type":"ContainerStarted","Data":"1b6f7c68b0e62a1c12390702edc07382e5ddd2a7f3a7a66a648fc21ae66a3d24"} Jul 10 20:37:54.061624 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.061585 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerStarted","Data":"cec64ee5924c748ea5fa3b48aee06166414d36f921f7c249ea91b7f6fb3c9780"} Jul 10 20:37:54.064710 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.064670 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zzgw" event={"ID":"cfa3b044-cb6f-41f8-b5d8-96fe2948004c","Type":"ContainerStarted","Data":"838ad1669865098fe3db021c2d003277362718202d2033d8d20df58b96348cca"} Jul 10 20:37:54.073069 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.072888 2585 generic.go:356] "Generic (PLEG): container finished" podID="ebc7c1f32995e2f09dba311721291d26" containerID="d42b7621cacce8fd817fd57cda059380627829298c638a11977195beff8d29f2" exitCode=0 Jul 10 20:37:54.073229 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.073206 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"ebc7c1f32995e2f09dba311721291d26","Type":"ContainerDied","Data":"d42b7621cacce8fd817fd57cda059380627829298c638a11977195beff8d29f2"} Jul 10 20:37:54.086398 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.086368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"aab2087575a5b19ff9008afc3f07598f8dfa4e8ab1c7b740fd1359db60a8c566"} Jul 10 20:37:54.089067 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.089010 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-194.ec2.internal" podStartSLOduration=3.088990241 podStartE2EDuration="3.088990241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:37:54.024183459 +0000 UTC m=+4.630940064" watchObservedRunningTime="2026-07-10 20:37:54.088990241 +0000 UTC m=+4.695746827" Jul 10 20:37:54.093177 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.093147 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" event={"ID":"88918d10-2931-4413-9080-2f6140a21cda","Type":"ContainerStarted","Data":"f25018f2e3d1d509c63c2885831def5d3223fd3a45536486522796ea8d42b6e9"} Jul 10 20:37:54.110620 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.110585 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rmqm6" event={"ID":"a989078a-a850-4db3-9042-d195a3e461e7","Type":"ContainerStarted","Data":"5c4a815b5df3f8d83c7ce468178ac5a8b7c6058171fddb6ff614cc14720cf74b"} Jul 10 20:37:54.124480 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.124438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dgdgm" event={"ID":"56ff23da-1f36-4fc2-b33e-230ba5252990","Type":"ContainerStarted","Data":"07ee687616580f4e2586229c2efe986973963c9272f795c0a76333d488bccecb"} Jul 10 20:37:54.136609 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.136572 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szhnj" event={"ID":"780f7dc2-8673-4890-bab2-2e0f6ef42c3e","Type":"ContainerStarted","Data":"4e77b5de223ebb116ecfa9e58a5ada7cac201ed7bb9c50389f0ea0e46e814fa2"} Jul 10 20:37:54.980436 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:54.979863 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:54.980436 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:54.980029 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:37:55.153547 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.153507 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" event={"ID":"ebc7c1f32995e2f09dba311721291d26","Type":"ContainerStarted","Data":"040557d19302ced4213cdc7682a236139bf4e9b395a1466ca82e285834c68a56"} Jul 10 20:37:55.457678 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.457584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:55.457863 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.457743 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:55.457863 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.457810 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:59.457792661 +0000 UTC m=+10.064549225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:55.558239 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.558197 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:55.558435 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.558399 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:55.558511 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.558468 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:59.558448125 +0000 UTC m=+10.165204691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:55.660435 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.659644 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:55.660435 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.659858 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:37:55.660435 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.659892 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:37:55.660435 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.659906 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:55.660435 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.659975 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:37:59.659954205 +0000 UTC m=+10.266710790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:55.982092 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.982056 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:55.982607 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.982198 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:37:55.982674 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:55.982629 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:55.982781 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:55.982729 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:37:56.979764 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:56.979723 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:56.979964 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:56.979866 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:37:57.981503 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:57.981403 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:57.981503 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:57.981472 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:57.981953 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:57.981567 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:37:57.981953 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:57.981726 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:37:58.979978 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:58.979940 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:58.980179 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:58.980092 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:37:59.491842 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:59.491309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:59.491842 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.491445 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:59.491842 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.491509 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:07.491490763 +0000 UTC m=+18.098247335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:37:59.591997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:59.591959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:37:59.592165 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.592128 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:59.592220 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.592195 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:07.592174494 +0000 UTC m=+18.198931056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:37:59.693382 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:59.693344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:59.693566 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.693521 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:37:59.693566 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.693541 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:37:59.693566 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.693554 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:59.693721 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.693614 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:07.693593151 +0000 UTC m=+18.300349725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:37:59.979837 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:59.979748 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:37:59.979978 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.979868 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:37:59.980195 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:37:59.980175 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:37:59.980342 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:37:59.980262 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:00.979630 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:00.979593 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:00.980080 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:00.979752 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:01.979310 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:01.979268 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:01.979491 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:01.979268 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:01.979491 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:01.979424 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:01.979597 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:01.979499 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:02.979972 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:02.979937 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:02.980457 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:02.980094 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:03.979945 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:03.979908 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:03.980107 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:03.979950 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:03.980107 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:03.980036 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:03.980411 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:03.980388 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:04.979482 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:04.979453 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:04.979654 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:04.979570 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:05.979532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:05.979496 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:05.979946 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:05.979637 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:05.979946 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:05.979688 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:05.979946 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:05.979785 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:06.979605 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:06.979559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:06.980083 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:06.979704 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:07.554884 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:07.554850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:07.555073 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.555033 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:38:07.555125 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.555116 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:23.555095044 +0000 UTC m=+34.161851622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:38:07.655585 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:07.655548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:07.655769 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.655725 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:38:07.655825 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.655813 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:23.655789643 +0000 UTC m=+34.262546217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:38:07.756284 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:07.756250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:07.756439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.756397 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:38:07.756439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.756416 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:38:07.756439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.756427 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:38:07.756577 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.756492 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:23.756473228 +0000 UTC m=+34.363229808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:38:07.979209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:07.979172 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:07.979209 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:07.979205 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:07.979415 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.979322 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:07.979467 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:07.979446 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:08.979611 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:08.979577 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:08.979975 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:08.979695 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:09.982838 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:09.982810 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:09.983306 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:09.982970 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:09.983306 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:09.983027 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:09.983306 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:09.983122 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:10.978958 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:10.978928 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:10.979272 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:10.979055 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:11.982381 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:11.982354 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:11.982843 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:11.982356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:11.982843 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:11.982471 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:11.982843 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:11.982555 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:12.979828 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:12.979652 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:12.980049 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:12.979908 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:13.193319 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.193293 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:38:13.193926 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.193713 2585 generic.go:356] "Generic (PLEG): container finished" podID="8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c" containerID="062aafa62631f6ac4c7ef825f30d487286fff9d6eafb15ad612b7d82cff8288c" exitCode=1 Jul 10 20:38:13.193926 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.193802 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"f55fb3e1b256ead01c7c9ec59d0ffb228de9b3df5e2ac2cd3028aecfafb24761"} Jul 10 20:38:13.193926 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.193884 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerDied","Data":"062aafa62631f6ac4c7ef825f30d487286fff9d6eafb15ad612b7d82cff8288c"} Jul 10 20:38:13.194074 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.193934 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"78904fb364ec42e7be1337412a0583dce87fb872d2a3fcd4af96e835381f5a27"} Jul 10 20:38:13.197219 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.197193 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" event={"ID":"88918d10-2931-4413-9080-2f6140a21cda","Type":"ContainerStarted","Data":"1621fcddf15b97c351993e23e64779db55a214d1cae264d440c4a6b70d9b74c8"} Jul 10 20:38:13.198895 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.198857 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rmqm6" event={"ID":"a989078a-a850-4db3-9042-d195a3e461e7","Type":"ContainerStarted","Data":"3b59e7bea45231e2a191ce42d9e57f072e22930ef2cb00be82a2eebd85a068a8"} Jul 10 20:38:13.202725 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.201655 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dgdgm" event={"ID":"56ff23da-1f36-4fc2-b33e-230ba5252990","Type":"ContainerStarted","Data":"efa0aac8b48be8a39600324160e1a2cd19ca0d4e6e82591def1ee4665d437dfc"} Jul 10 20:38:13.205022 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.204996 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6xrm" event={"ID":"cb239305-035b-4c95-bda0-aabdb6f65431","Type":"ContainerStarted","Data":"b4028b20bd34378b599016cf584fdf2246881b6608192a0e5a8025fc363e081d"} Jul 10 20:38:13.208085 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.208060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blr66" event={"ID":"83624a0b-c4ce-4308-bcb4-bcae87df2139","Type":"ContainerStarted","Data":"4f0aa012a1d1bd0459bc1ee0929391b2b1dcf7e5f6b773b6ad72c07fb9250889"} Jul 10 20:38:13.210169 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.210051 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="11e0b9d2836674fbe4af7af208e3683d21b311d1cb7026bf73a7cb98d2f936da" exitCode=0 Jul 10 20:38:13.210242 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.210211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"11e0b9d2836674fbe4af7af208e3683d21b311d1cb7026bf73a7cb98d2f936da"} Jul 10 20:38:13.212083 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.212057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zzgw" event={"ID":"cfa3b044-cb6f-41f8-b5d8-96fe2948004c","Type":"ContainerStarted","Data":"a0a589ff5270e42418b684de6f44efb442d88e35bb247d9d4a880d5fddff4052"} Jul 10 20:38:13.214328 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.214290 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-194.ec2.internal" podStartSLOduration=22.214277674 podStartE2EDuration="22.214277674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:37:55.169792936 +0000 UTC m=+5.776549539" watchObservedRunningTime="2026-07-10 20:38:13.214277674 +0000 UTC m=+23.821034256" Jul 10 20:38:13.214489 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.214471 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rmqm6" podStartSLOduration=12.256378502 podStartE2EDuration="23.214467352s" totalImagesPullingTime="10.95808885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:13.214261721 +0000 UTC m=+23.821018322" watchObservedRunningTime="2026-07-10 20:38:13.214467352 +0000 UTC m=+23.821223973" Jul 10 20:38:13.229519 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.229479 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v6xrm" podStartSLOduration=3.822017169 podStartE2EDuration="23.229464665s" totalImagesPullingTime="19.407447496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:13.228981717 +0000 UTC m=+23.835738300" watchObservedRunningTime="2026-07-10 20:38:13.229464665 +0000 UTC m=+23.836221247" Jul 10 20:38:13.242527 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.242485 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-blr66" podStartSLOduration=3.793766392 podStartE2EDuration="23.242471304s" totalImagesPullingTime="19.448704912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:13.242273992 +0000 UTC m=+23.849030603" watchObservedRunningTime="2026-07-10 20:38:13.242471304 +0000 UTC m=+23.849227882" Jul 10 20:38:13.255706 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.255526 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9zzgw" podStartSLOduration=3.840821665 podStartE2EDuration="23.255507808s" totalImagesPullingTime="19.414686143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:13.255006791 +0000 UTC m=+23.861763375" watchObservedRunningTime="2026-07-10 20:38:13.255507808 +0000 UTC m=+23.862264390" Jul 10 20:38:13.271430 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.271391 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dgdgm" podStartSLOduration=3.8279160020000003 podStartE2EDuration="23.271375964s" totalImagesPullingTime="19.443459962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:13.271329221 +0000 UTC m=+23.878085802" watchObservedRunningTime="2026-07-10 20:38:13.271375964 +0000 UTC m=+23.878132546" Jul 10 20:38:13.979951 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.979826 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:13.980025 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:13.979962 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:13.980081 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:13.980021 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:13.980154 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:13.980130 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:14.035266 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.035223 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 20:38:14.218460 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.218427 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:38:14.219003 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.218858 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"e5f40f3b82e3f7633c6eafd3e04aed40d3c0491d5afb55b898d1c252473269b3"} Jul 10 20:38:14.219003 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.218949 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"4fc896e36135c4182418c75880e9397394cbeafc42b1db3ac2427d351ac42655"} Jul 10 20:38:14.219003 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.218988 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"a66f3006ea59c882065d3b75db759162ee3b1ad315f04f78dcac701d03dd1dde"} Jul 10 20:38:14.220979 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.220954 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" event={"ID":"88918d10-2931-4413-9080-2f6140a21cda","Type":"ContainerStarted","Data":"09dd34ed74b87cfb94f8d853feac860248113579b5f42cfa7d1c6fd8920fe683"} Jul 10 20:38:14.222532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.222505 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szhnj" event={"ID":"780f7dc2-8673-4890-bab2-2e0f6ef42c3e","Type":"ContainerStarted","Data":"6ed8c6bcec2b2e6853d45ca4ef9ea21af9c6404629d776d74e1d799edb87ba85"} Jul 10 20:38:14.236807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.236767 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-szhnj" podStartSLOduration=4.800743709 podStartE2EDuration="24.2367535s" totalImagesPullingTime="19.436009791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:14.236373374 +0000 UTC m=+24.843129983" watchObservedRunningTime="2026-07-10 20:38:14.2367535 +0000 UTC m=+24.843510082" Jul 10 20:38:14.889757 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.889649 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T20:38:14.03526213Z","UUID":"0158e7e4-3d3a-4248-82ff-86ef965aea72","Handler":null,"Name":"","Endpoint":""} Jul 10 20:38:14.891309 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.891290 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 20:38:14.891309 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.891324 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 20:38:14.979149 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:14.979116 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:14.979323 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:14.979270 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:15.226913 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:15.226876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" event={"ID":"88918d10-2931-4413-9080-2f6140a21cda","Type":"ContainerStarted","Data":"817db95ed1ca1102df89a2cbe54fda17f1113397d9b7707cac4406ff78dcfaed"} Jul 10 20:38:15.242183 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:15.242136 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6jdz" podStartSLOduration=3.907441594 podStartE2EDuration="25.242119102s" totalImagesPullingTime="21.334677508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:15.241424675 +0000 UTC m=+25.848181257" watchObservedRunningTime="2026-07-10 20:38:15.242119102 +0000 UTC m=+25.848875663" Jul 10 20:38:15.979693 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:15.979651 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:15.979874 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:15.979780 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:15.979874 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:15.979830 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:15.979990 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:15.979944 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:16.223248 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.223205 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:38:16.224051 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.224029 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:38:16.232311 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.232068 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:38:16.232722 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.232699 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"3802d42acb2e55f726e681e2000c2d157ca33a4df6b976202da21e81b184ebe7"} Jul 10 20:38:16.233171 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.233146 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:38:16.233613 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.233592 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rmqm6" Jul 10 20:38:16.979616 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:16.979589 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:16.979773 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:16.979710 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:17.979414 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:17.979162 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:17.980073 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:17.979258 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:17.980073 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:17.979508 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:17.980073 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:17.979572 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:18.239040 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.239003 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="c1d91e9f5f287808ba779adf9ccd63edf9dbbb335a650e9ec4c8b93eae153310" exitCode=0 Jul 10 20:38:18.239224 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.239089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"c1d91e9f5f287808ba779adf9ccd63edf9dbbb335a650e9ec4c8b93eae153310"} Jul 10 20:38:18.242727 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.242708 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:38:18.243090 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.243068 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"0e14189fc3117d1a8eb7360651e5bae78ce4a85429fb6e3ab75307cbdd7244e6"} Jul 10 20:38:18.243501 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.243487 2585 scope.go:117] "RemoveContainer" containerID="062aafa62631f6ac4c7ef825f30d487286fff9d6eafb15ad612b7d82cff8288c" Jul 10 20:38:18.979739 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:18.979696 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:18.980170 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:18.979820 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:19.247372 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.247299 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="1044d6ddddb196cf9a95f279448fa28d4d6929d4b350bf084d692aa29a312a1e" exitCode=0 Jul 10 20:38:19.247538 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.247381 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"1044d6ddddb196cf9a95f279448fa28d4d6929d4b350bf084d692aa29a312a1e"} Jul 10 20:38:19.251387 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.251369 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:38:19.251707 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.251682 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" event={"ID":"8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c","Type":"ContainerStarted","Data":"6e8b427d5c22ff0d2ee9bcf9b05cca688b9321fd60c3824c6806a09893d325e8"} Jul 10 20:38:19.251894 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.251878 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:19.251987 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.251949 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:19.252050 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.252002 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:19.270164 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.270115 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:19.273712 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.273689 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:19.302286 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.302202 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" podStartSLOduration=9.822729335 podStartE2EDuration="29.302181452s" totalImagesPullingTime="19.479452117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:19.301282088 +0000 UTC m=+29.908038692" watchObservedRunningTime="2026-07-10 20:38:19.302181452 +0000 UTC m=+29.908938053" Jul 10 20:38:19.597916 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.597827 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdr8l"] Jul 10 20:38:19.598080 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.597982 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:19.598149 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:19.598096 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:19.598531 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.598507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2tc"] Jul 10 20:38:19.598634 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.598595 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:19.598677 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:19.598659 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:19.600791 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.600767 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vnrsl"] Jul 10 20:38:19.600901 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:19.600852 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:19.600963 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:19.600947 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:20.256372 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:20.256169 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="10a21e0589ef4f74ff79a4547839a099910e7995c751c1e3399f374020c76039" exitCode=0 Jul 10 20:38:20.256372 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:20.256269 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"10a21e0589ef4f74ff79a4547839a099910e7995c751c1e3399f374020c76039"} Jul 10 20:38:20.979707 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:20.979673 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:20.979707 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:20.979635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:20.979707 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:20.979705 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:20.979963 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:20.979870 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:20.980073 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:20.980047 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:20.980147 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:20.980125 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:22.979317 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:22.979282 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:22.979317 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:22.979304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:22.979788 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:22.979290 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:22.979788 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:22.979394 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:22.979788 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:22.979476 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:22.979788 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:22.979554 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:23.580509 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:23.580465 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:23.580715 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.580627 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:38:23.580780 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.580716 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret podName:1842cdd3-d259-4106-9d65-618540a99480 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:55.580695708 +0000 UTC m=+66.187452268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret") pod "global-pull-secret-syncer-xdr8l" (UID: "1842cdd3-d259-4106-9d65-618540a99480") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:38:23.680959 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:23.680922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:23.681113 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.681074 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:38:23.681183 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.681142 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:55.681120816 +0000 UTC m=+66.287877379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:38:23.781870 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:23.781825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:23.782038 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.781972 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:38:23.782038 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.781995 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:38:23.782038 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.782007 2585 projected.go:196] Error preparing data for projected volume kube-api-access-wm9ng for pod openshift-network-diagnostics/network-check-target-zr2tc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:38:23.782218 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:23.782071 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng podName:eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:55.782052684 +0000 UTC m=+66.388809246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wm9ng" (UniqueName: "kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng") pod "network-check-target-zr2tc" (UID: "eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:38:24.979934 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:24.979902 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:24.980439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:24.979940 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:24.980439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:24.979946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:24.980439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:24.980015 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr8l" podUID="1842cdd3-d259-4106-9d65-618540a99480" Jul 10 20:38:24.980439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:24.980096 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2tc" podUID="eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4" Jul 10 20:38:24.980439 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:24.980168 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:38:25.271843 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:25.271769 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="91b51e51b7447e6c9b5718ffd35ff8c863510bc23f58feafc3d47a1ae9dc8b05" exitCode=0 Jul 10 20:38:25.271843 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:25.271810 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"91b51e51b7447e6c9b5718ffd35ff8c863510bc23f58feafc3d47a1ae9dc8b05"} Jul 10 20:38:26.277130 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.277087 2585 generic.go:356] "Generic (PLEG): container finished" podID="a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0" containerID="f8754ee87fbc9e39e6314269b4f4556e80b16a5f446718b4ce9aea7028bcacdd" exitCode=0 Jul 10 20:38:26.277601 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.277161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerDied","Data":"f8754ee87fbc9e39e6314269b4f4556e80b16a5f446718b4ce9aea7028bcacdd"} Jul 10 20:38:26.719159 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.719130 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-194.ec2.internal" event="NodeReady" Jul 10 20:38:26.719340 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.719293 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 20:38:26.761900 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.761866 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n66mb"] Jul 10 20:38:26.784011 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.783982 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2kbcj"] Jul 10 20:38:26.784170 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.784113 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:26.785810 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.785788 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 20:38:26.785945 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.785829 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 20:38:26.785945 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.785840 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h798g\"" Jul 10 20:38:26.806499 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.806469 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n66mb"] Jul 10 20:38:26.806499 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.806504 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kbcj"] Jul 10 20:38:26.806684 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.806581 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:26.809985 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.809952 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 20:38:26.810242 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.810212 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 20:38:26.810312 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.810269 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 20:38:26.810389 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.810374 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8wsvp\"" Jul 10 20:38:26.906589 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5621c9-1df4-4e9d-87a1-335f497c90bc-tmp-dir\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:26.906780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rssvk\" (UniqueName: \"kubernetes.io/projected/f9d1ff0b-f1ed-410b-9655-fe196396be09-kube-api-access-rssvk\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:26.906780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906660 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9d1ff0b-f1ed-410b-9655-fe196396be09-cert\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:26.906780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906689 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:26.906780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5621c9-1df4-4e9d-87a1-335f497c90bc-config-volume\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:26.906780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.906771 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcn9\" (UniqueName: \"kubernetes.io/projected/ce5621c9-1df4-4e9d-87a1-335f497c90bc-kube-api-access-fzcn9\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:26.979785 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.979699 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:26.979785 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.979747 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:26.979785 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.979707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:26.981766 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.981741 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:38:26.981878 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.981759 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:38:26.981878 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.981748 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6zkh\"" Jul 10 20:38:26.981982 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.981902 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:38:26.981982 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.981937 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggkq4\"" Jul 10 20:38:26.982073 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:26.982024 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:38:27.007520 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcn9\" (UniqueName: \"kubernetes.io/projected/ce5621c9-1df4-4e9d-87a1-335f497c90bc-kube-api-access-fzcn9\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.007670 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5621c9-1df4-4e9d-87a1-335f497c90bc-tmp-dir\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.007670 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rssvk\" (UniqueName: \"kubernetes.io/projected/f9d1ff0b-f1ed-410b-9655-fe196396be09-kube-api-access-rssvk\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:27.007670 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9d1ff0b-f1ed-410b-9655-fe196396be09-cert\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:27.007670 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.007670 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.007633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5621c9-1df4-4e9d-87a1-335f497c90bc-config-volume\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.007938 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:27.007724 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:27.007938 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:27.007774 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:27.507756594 +0000 UTC m=+38.114513154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:27.008036 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.008002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5621c9-1df4-4e9d-87a1-335f497c90bc-tmp-dir\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.008201 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.008183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5621c9-1df4-4e9d-87a1-335f497c90bc-config-volume\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.012197 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.012164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9d1ff0b-f1ed-410b-9655-fe196396be09-cert\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:27.014884 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.014864 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rssvk\" (UniqueName: \"kubernetes.io/projected/f9d1ff0b-f1ed-410b-9655-fe196396be09-kube-api-access-rssvk\") pod \"ingress-canary-2kbcj\" (UID: \"f9d1ff0b-f1ed-410b-9655-fe196396be09\") " pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:27.015070 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.015050 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcn9\" (UniqueName: \"kubernetes.io/projected/ce5621c9-1df4-4e9d-87a1-335f497c90bc-kube-api-access-fzcn9\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.115688 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.115654 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kbcj" Jul 10 20:38:27.287628 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.287447 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dl78j" event={"ID":"a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0","Type":"ContainerStarted","Data":"58c90c7fd514817806510404c17bea15ea15fe85a599b64b70ffc8ce136992e0"} Jul 10 20:38:27.302765 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.302730 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kbcj"] Jul 10 20:38:27.306648 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:38:27.306622 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d1ff0b_f1ed_410b_9655_fe196396be09.slice/crio-6fe10ceb89ede1757bf460e0cf298606f3ce5cc288a105071029b65c10affb4a WatchSource:0}: Error finding container 6fe10ceb89ede1757bf460e0cf298606f3ce5cc288a105071029b65c10affb4a: Status 404 returned error can't find the container with id 6fe10ceb89ede1757bf460e0cf298606f3ce5cc288a105071029b65c10affb4a Jul 10 20:38:27.310427 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.310384 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dl78j" podStartSLOduration=7.57090516 podStartE2EDuration="37.310365198s" totalImagesPullingTime="29.739460038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:27.309690352 +0000 UTC m=+37.916446979" watchObservedRunningTime="2026-07-10 20:38:27.310365198 +0000 UTC m=+37.917121779" Jul 10 20:38:27.511821 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:27.511732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:27.511965 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:27.511894 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:27.512012 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:27.511969 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:28.511950188 +0000 UTC m=+39.118706749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:28.291761 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:28.291714 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kbcj" event={"ID":"f9d1ff0b-f1ed-410b-9655-fe196396be09","Type":"ContainerStarted","Data":"6fe10ceb89ede1757bf460e0cf298606f3ce5cc288a105071029b65c10affb4a"} Jul 10 20:38:28.518685 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:28.518646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:28.518861 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:28.518824 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:28.518920 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:28.518908 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:30.518886602 +0000 UTC m=+41.125643164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:30.299145 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:30.298880 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kbcj" event={"ID":"f9d1ff0b-f1ed-410b-9655-fe196396be09","Type":"ContainerStarted","Data":"289a189e8813bf142f2427a5e72ee40bc9fa06f4b7ceb517d4f77cf971993b21"} Jul 10 20:38:30.314578 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:30.314517 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2kbcj" podStartSLOduration=1.596252684 podStartE2EDuration="4.314493628s" totalImagesPullingTime="2.718240944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:30.312994106 +0000 UTC m=+40.919750711" watchObservedRunningTime="2026-07-10 20:38:30.314493628 +0000 UTC m=+40.921250211" Jul 10 20:38:30.532301 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:30.532215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:30.532521 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:30.532352 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:30.532521 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:30.532431 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:34.532410758 +0000 UTC m=+45.139167332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:34.559778 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:34.559720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:34.560306 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:34.559894 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:34.560306 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:34.559960 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:42.559944586 +0000 UTC m=+53.166701146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:42.615342 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:42.615300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:42.615856 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:42.615462 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:42.615856 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:42.615528 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:38:58.615509756 +0000 UTC m=+69.222266315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:38:47.490992 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.490960 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn"] Jul 10 20:38:47.492832 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.492815 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.495180 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.495149 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 20:38:47.495348 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.495201 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 20:38:47.495348 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.495270 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 20:38:47.495348 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.495327 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 20:38:47.505437 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.505414 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn"] Jul 10 20:38:47.519249 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.519209 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65"] Jul 10 20:38:47.521172 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.521153 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.523032 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.523008 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 20:38:47.523158 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.523037 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 20:38:47.523158 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.523062 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 20:38:47.523385 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.523370 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 20:38:47.533726 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.533707 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65"] Jul 10 20:38:47.558181 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltx85\" (UniqueName: \"kubernetes.io/projected/6442f444-608c-4596-8ba8-cadf0a894f37-kube-api-access-ltx85\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558181 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eed602c5-65b3-43f4-ae22-1564f3490911-tmp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.558353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558353 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558475 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eed602c5-65b3-43f4-ae22-1564f3490911-klusterlet-config\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.558475 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558418 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6442f444-608c-4596-8ba8-cadf0a894f37-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.558475 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.558434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfkp\" (UniqueName: \"kubernetes.io/projected/eed602c5-65b3-43f4-ae22-1564f3490911-kube-api-access-frfkp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.659609 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6442f444-608c-4596-8ba8-cadf0a894f37-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.659609 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfkp\" (UniqueName: \"kubernetes.io/projected/eed602c5-65b3-43f4-ae22-1564f3490911-kube-api-access-frfkp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltx85\" (UniqueName: \"kubernetes.io/projected/6442f444-608c-4596-8ba8-cadf0a894f37-kube-api-access-ltx85\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659650 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eed602c5-65b3-43f4-ae22-1564f3490911-tmp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659674 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659703 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.659783 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.660087 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.659812 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eed602c5-65b3-43f4-ae22-1564f3490911-klusterlet-config\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.660144 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.660085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eed602c5-65b3-43f4-ae22-1564f3490911-tmp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.660448 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.660412 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6442f444-608c-4596-8ba8-cadf0a894f37-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.664067 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.664042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-ca\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.664176 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.664113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.664176 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.664135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.664294 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.664197 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eed602c5-65b3-43f4-ae22-1564f3490911-klusterlet-config\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.664294 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.664212 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6442f444-608c-4596-8ba8-cadf0a894f37-hub\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.666476 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.666454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfkp\" (UniqueName: \"kubernetes.io/projected/eed602c5-65b3-43f4-ae22-1564f3490911-kube-api-access-frfkp\") pod \"klusterlet-addon-workmgr-798c5659ff-8tbsn\" (UID: \"eed602c5-65b3-43f4-ae22-1564f3490911\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.666804 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.666783 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltx85\" (UniqueName: \"kubernetes.io/projected/6442f444-608c-4596-8ba8-cadf0a894f37-kube-api-access-ltx85\") pod \"cluster-proxy-proxy-agent-7bfccbd794-s4c65\" (UID: \"6442f444-608c-4596-8ba8-cadf0a894f37\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.801774 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.801684 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:47.840146 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.840101 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:38:47.935660 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.935363 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn"] Jul 10 20:38:47.941029 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:38:47.940992 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed602c5_65b3_43f4_ae22_1564f3490911.slice/crio-24ecc4247e589f59cdeb1bc6cc75c9161b2a8e3e6f006a787d6e6653686245ad WatchSource:0}: Error finding container 24ecc4247e589f59cdeb1bc6cc75c9161b2a8e3e6f006a787d6e6653686245ad: Status 404 returned error can't find the container with id 24ecc4247e589f59cdeb1bc6cc75c9161b2a8e3e6f006a787d6e6653686245ad Jul 10 20:38:47.986030 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:47.985996 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65"] Jul 10 20:38:47.989465 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:38:47.989437 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6442f444_608c_4596_8ba8_cadf0a894f37.slice/crio-0c2e853f93b3c12391a2dc53666529ae772b37d2511928822ece113987a16e96 WatchSource:0}: Error finding container 0c2e853f93b3c12391a2dc53666529ae772b37d2511928822ece113987a16e96: Status 404 returned error can't find the container with id 0c2e853f93b3c12391a2dc53666529ae772b37d2511928822ece113987a16e96 Jul 10 20:38:48.340961 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:48.340917 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" event={"ID":"eed602c5-65b3-43f4-ae22-1564f3490911","Type":"ContainerStarted","Data":"24ecc4247e589f59cdeb1bc6cc75c9161b2a8e3e6f006a787d6e6653686245ad"} Jul 10 20:38:48.341872 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:48.341849 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerStarted","Data":"0c2e853f93b3c12391a2dc53666529ae772b37d2511928822ece113987a16e96"} Jul 10 20:38:51.270739 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:51.270706 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxw5l" Jul 10 20:38:53.357992 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.357940 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" event={"ID":"eed602c5-65b3-43f4-ae22-1564f3490911","Type":"ContainerStarted","Data":"e60f4c0cb5eb270cd87ee959fe25b99d496c07ac24bccfd8f2f8194f38a328ac"} Jul 10 20:38:53.358467 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.358013 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:53.359787 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.359766 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:38:53.360195 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.360170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerStarted","Data":"192a42b457db27e1fbb8fb5d6b6d55b9e25edcffa7b1a17a0d23286c48ff3cdd"} Jul 10 20:38:53.360310 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.360201 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerStarted","Data":"00f2305b8678eb6b294d8f34d249bf20b00c391aaa8ab04a252de2f4d3ee2350"} Jul 10 20:38:53.360310 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.360216 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerStarted","Data":"d9fc4f84195676e11852c30fed6460c33eaf66901065210a9bbb734bbd23aac7"} Jul 10 20:38:53.375510 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.375466 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" podStartSLOduration=1.174976194 podStartE2EDuration="6.375451727s" totalImagesPullingTime="5.200475533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:53.374945456 +0000 UTC m=+63.981702038" watchObservedRunningTime="2026-07-10 20:38:53.375451727 +0000 UTC m=+63.982208345" Jul 10 20:38:53.408715 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:53.408608 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" podStartSLOduration=1.257313406 podStartE2EDuration="6.408592227s" totalImagesPullingTime="5.151278821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:53.407846721 +0000 UTC m=+64.014603303" watchObservedRunningTime="2026-07-10 20:38:53.408592227 +0000 UTC m=+64.015348810" Jul 10 20:38:55.626483 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.626439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:55.628611 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.628593 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:38:55.639258 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.639208 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1842cdd3-d259-4106-9d65-618540a99480-original-pull-secret\") pod \"global-pull-secret-syncer-xdr8l\" (UID: \"1842cdd3-d259-4106-9d65-618540a99480\") " pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:55.727623 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.727585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:38:55.729664 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.729646 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:38:55.738037 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:55.738014 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:38:55.738101 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:55.738087 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:39:59.738064145 +0000 UTC m=+130.344820705 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : secret "metrics-daemon-secret" not found Jul 10 20:38:55.800348 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.800307 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr8l" Jul 10 20:38:55.828739 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.828711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:55.830916 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.830895 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:38:55.841245 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.841207 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:38:55.853211 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.853164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9ng\" (UniqueName: \"kubernetes.io/projected/eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4-kube-api-access-wm9ng\") pod \"network-check-target-zr2tc\" (UID: \"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4\") " pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:55.943762 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:55.943733 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdr8l"] Jul 10 20:38:55.946957 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:38:55.946925 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1842cdd3_d259_4106_9d65_618540a99480.slice/crio-ba7b887993be0297f639ea927bd30599e61766747bf77fc9651a8dacf597c2b6 WatchSource:0}: Error finding container ba7b887993be0297f639ea927bd30599e61766747bf77fc9651a8dacf597c2b6: Status 404 returned error can't find the container with id ba7b887993be0297f639ea927bd30599e61766747bf77fc9651a8dacf597c2b6 Jul 10 20:38:56.091473 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:56.091440 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggkq4\"" Jul 10 20:38:56.100081 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:56.100055 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:38:56.252541 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:56.252510 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2tc"] Jul 10 20:38:56.254788 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:38:56.254764 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7ad72d_c31b_41ed_9e6a_c8e048f3cec4.slice/crio-2943ce365de1f43229f58326ed9dfa46a42182e25c939355b6576fed6965a020 WatchSource:0}: Error finding container 2943ce365de1f43229f58326ed9dfa46a42182e25c939355b6576fed6965a020: Status 404 returned error can't find the container with id 2943ce365de1f43229f58326ed9dfa46a42182e25c939355b6576fed6965a020 Jul 10 20:38:56.370357 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:56.370313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2tc" event={"ID":"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4","Type":"ContainerStarted","Data":"2943ce365de1f43229f58326ed9dfa46a42182e25c939355b6576fed6965a020"} Jul 10 20:38:56.371540 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:56.371502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdr8l" event={"ID":"1842cdd3-d259-4106-9d65-618540a99480","Type":"ContainerStarted","Data":"ba7b887993be0297f639ea927bd30599e61766747bf77fc9651a8dacf597c2b6"} Jul 10 20:38:58.652867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:38:58.652828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:38:58.653399 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:58.652960 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:38:58.653399 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:38:58.653016 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:39:30.653002346 +0000 UTC m=+101.259758909 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:39:01.388455 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:01.388419 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdr8l" event={"ID":"1842cdd3-d259-4106-9d65-618540a99480","Type":"ContainerStarted","Data":"2031331b8a780ec2b5fa47aea6506b26841267255d792bf8b8c7341b2fdeb75c"} Jul 10 20:39:01.389984 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:01.389959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2tc" event={"ID":"eb7ad72d-c31b-41ed-9e6a-c8e048f3cec4","Type":"ContainerStarted","Data":"f8cbb2e7e2fc96bbf85d4a3a78ccfd76625e4508cde7c703a6730e8eefde6354"} Jul 10 20:39:01.390109 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:01.390096 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:39:01.415523 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:01.415482 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xdr8l" podStartSLOduration=66.804030061 podStartE2EDuration="1m11.415468633s" totalImagesPullingTime="4.611438572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:01.401494399 +0000 UTC m=+72.008250981" watchObservedRunningTime="2026-07-10 20:39:01.415468633 +0000 UTC m=+72.022225214" Jul 10 20:39:01.415644 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:01.415553 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zr2tc" podStartSLOduration=67.118071332 podStartE2EDuration="1m11.415549199s" totalImagesPullingTime="4.297477867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:01.414661454 +0000 UTC m=+72.021418044" watchObservedRunningTime="2026-07-10 20:39:01.415549199 +0000 UTC m=+72.022305759" Jul 10 20:39:30.686675 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:30.686631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:39:30.687097 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:39:30.686809 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:39:30.687097 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:39:30.686877 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:40:34.686858091 +0000 UTC m=+165.293614654 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:39:32.395137 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:32.395110 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zr2tc" Jul 10 20:39:59.796629 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:39:59.796588 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:39:59.797104 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:39:59.796742 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:39:59.797104 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:39:59.796818 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:01.796801781 +0000 UTC m=+252.403558341 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : secret "metrics-daemon-secret" not found Jul 10 20:40:29.793113 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:40:29.793077 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n66mb" podUID="ce5621c9-1df4-4e9d-87a1-335f497c90bc" Jul 10 20:40:29.995220 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:40:29.995177 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:40:30.648992 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:30.648963 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n66mb" Jul 10 20:40:34.749364 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:34.749326 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:40:34.749831 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:40:34.749489 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:40:34.749831 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:40:34.749557 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls podName:ce5621c9-1df4-4e9d-87a1-335f497c90bc nodeName:}" failed. No retries permitted until 2026-07-10 20:42:36.749539772 +0000 UTC m=+287.356296336 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls") pod "dns-default-n66mb" (UID: "ce5621c9-1df4-4e9d-87a1-335f497c90bc") : secret "dns-default-metrics-tls" not found Jul 10 20:40:44.979300 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:44.979187 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:40:53.359387 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:53.359331 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" podUID="eed602c5-65b3-43f4-ae22-1564f3490911" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jul 10 20:40:53.715112 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:53.715082 2585 generic.go:356] "Generic (PLEG): container finished" podID="eed602c5-65b3-43f4-ae22-1564f3490911" containerID="e60f4c0cb5eb270cd87ee959fe25b99d496c07ac24bccfd8f2f8194f38a328ac" exitCode=1 Jul 10 20:40:53.715393 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:53.715128 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" event={"ID":"eed602c5-65b3-43f4-ae22-1564f3490911","Type":"ContainerDied","Data":"e60f4c0cb5eb270cd87ee959fe25b99d496c07ac24bccfd8f2f8194f38a328ac"} Jul 10 20:40:53.715487 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:53.715428 2585 scope.go:117] "RemoveContainer" containerID="e60f4c0cb5eb270cd87ee959fe25b99d496c07ac24bccfd8f2f8194f38a328ac" Jul 10 20:40:54.719097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:54.719056 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" event={"ID":"eed602c5-65b3-43f4-ae22-1564f3490911","Type":"ContainerStarted","Data":"c07b4c5a915b547ff67ea98d41a4c3fafd1d9e2b7757ea3fd4141cfe20c3d632"} Jul 10 20:40:54.719601 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:54.719395 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:40:54.719974 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:40:54.719960 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-798c5659ff-8tbsn" Jul 10 20:41:59.070118 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.070085 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2r288"] Jul 10 20:41:59.072105 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.072090 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.074096 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.074073 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 20:41:59.074644 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.074626 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 20:41:59.074867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.074853 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt5hx\"" Jul 10 20:41:59.075077 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.075063 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 20:41:59.075122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.075081 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 20:41:59.092655 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.092626 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2r288"] Jul 10 20:41:59.135449 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.135405 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.135651 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.135457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fee6004-aff7-41a0-822b-bb6bae06d810-crio-socket\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.135651 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.135535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.135651 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.135564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fee6004-aff7-41a0-822b-bb6bae06d810-data-volume\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.135651 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.135599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvh4\" (UniqueName: \"kubernetes.io/projected/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-api-access-klvh4\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236556 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236556 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fee6004-aff7-41a0-822b-bb6bae06d810-crio-socket\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fee6004-aff7-41a0-822b-bb6bae06d810-data-volume\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvh4\" (UniqueName: \"kubernetes.io/projected/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-api-access-klvh4\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.236676 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fee6004-aff7-41a0-822b-bb6bae06d810-crio-socket\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.236792 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.236686 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:41:59.236996 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.236803 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:41:59.736779575 +0000 UTC m=+250.343536135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : configmap "kube-rbac-proxy" not found Jul 10 20:41:59.236996 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.236848 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:41:59.736829664 +0000 UTC m=+250.343586229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:41:59.236996 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.236907 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fee6004-aff7-41a0-822b-bb6bae06d810-data-volume\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.246207 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.246180 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvh4\" (UniqueName: \"kubernetes.io/projected/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-api-access-klvh4\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.741029 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.740989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.741214 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:41:59.741050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:41:59.741214 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.741125 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:41:59.741214 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.741143 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:41:59.741214 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.741185 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:00.74117101 +0000 UTC m=+251.347927570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : configmap "kube-rbac-proxy" not found Jul 10 20:41:59.741214 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:41:59.741197 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:00.741192081 +0000 UTC m=+251.347948642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:42:00.746724 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:00.746671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:00.747150 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:00.746781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:00.747150 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:00.746805 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:42:00.747150 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:00.746878 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:02.746858538 +0000 UTC m=+253.353615115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : configmap "kube-rbac-proxy" not found Jul 10 20:42:00.747150 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:00.746887 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:42:00.747150 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:00.746930 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:02.746920181 +0000 UTC m=+253.353676740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:42:01.839724 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:01.839692 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zzgw_cfa3b044-cb6f-41f8-b5d8-96fe2948004c/dns-node-resolver/0.log" Jul 10 20:42:01.855135 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:01.855106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:42:01.855308 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:01.855290 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:42:01.855381 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:01.855364 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs podName:f525f08f-e462-48e9-8530-85773208a545 nodeName:}" failed. No retries permitted until 2026-07-10 20:44:03.855343712 +0000 UTC m=+374.462100273 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs") pod "network-metrics-daemon-vnrsl" (UID: "f525f08f-e462-48e9-8530-85773208a545") : secret "metrics-daemon-secret" not found Jul 10 20:42:02.760076 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:02.760042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:02.760322 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:02.760108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:02.760322 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:02.760172 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:42:02.760322 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:02.760208 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:42:02.760322 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:02.760259 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:06.760222452 +0000 UTC m=+257.366979012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : configmap "kube-rbac-proxy" not found Jul 10 20:42:02.760322 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:02.760282 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:06.760270306 +0000 UTC m=+257.367026867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:42:02.841331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:02.841302 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6xrm_cb239305-035b-4c95-bda0-aabdb6f65431/node-ca/0.log" Jul 10 20:42:03.440508 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:03.440480 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2kbcj_f9d1ff0b-f1ed-410b-9655-fe196396be09/serve-healthcheck-canary/0.log" Jul 10 20:42:06.786695 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:06.786653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:06.787159 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:06.786708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:06.787159 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:06.786786 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:42:06.787159 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:06.786799 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:42:06.787159 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:06.786845 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:14.786829919 +0000 UTC m=+265.393586481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : configmap "kube-rbac-proxy" not found Jul 10 20:42:06.787159 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:06.786861 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:14.786854833 +0000 UTC m=+265.393611393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:42:14.847129 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:14.847038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:14.847129 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:14.847097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:14.847564 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:14.847201 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:42:14.847564 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:14.847292 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls podName:6fee6004-aff7-41a0-822b-bb6bae06d810 nodeName:}" failed. No retries permitted until 2026-07-10 20:42:30.847274733 +0000 UTC m=+281.454031292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2r288" (UID: "6fee6004-aff7-41a0-822b-bb6bae06d810") : secret "insights-runtime-extractor-tls" not found Jul 10 20:42:14.847641 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:14.847613 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fee6004-aff7-41a0-822b-bb6bae06d810-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:21.644254 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.644194 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84fb596665-kl4sk"] Jul 10 20:42:21.646220 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.646198 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.649140 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.649114 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 20:42:21.650409 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.650390 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 20:42:21.650746 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.650717 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dtnv5\"" Jul 10 20:42:21.650837 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.650798 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 20:42:21.655306 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.655288 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 20:42:21.664899 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.664870 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb596665-kl4sk"] Jul 10 20:42:21.695159 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695124 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-installation-pull-secrets\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695368 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-tls\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695368 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-image-registry-private-configuration\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695368 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695326 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-bound-sa-token\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg88z\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-kube-api-access-vg88z\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-ca-trust-extracted\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695437 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-certificates\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.695532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.695497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-trusted-ca\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-tls\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-image-registry-private-configuration\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796219 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-bound-sa-token\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg88z\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-kube-api-access-vg88z\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796545 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-ca-trust-extracted\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796545 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-certificates\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796637 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796552 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-trusted-ca\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796637 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-installation-pull-secrets\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.796936 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.796909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-ca-trust-extracted\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.797346 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.797325 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-certificates\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.797716 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.797692 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-trusted-ca\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.798863 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.798824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-image-registry-private-configuration\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.798955 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.798892 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-installation-pull-secrets\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.798997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.798974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-registry-tls\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.805634 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.805608 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-bound-sa-token\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.805884 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.805866 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg88z\" (UniqueName: \"kubernetes.io/projected/d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5-kube-api-access-vg88z\") pod \"image-registry-84fb596665-kl4sk\" (UID: \"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5\") " pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:21.955555 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:21.955524 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:22.079934 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:22.079900 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb596665-kl4sk"] Jul 10 20:42:22.084325 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:42:22.084296 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eb31fc_c5b5_48ce_8458_3cd8ed838cb5.slice/crio-24e3edd31763bed9d120fd2bae76fdc17bf1b9570e4e836d96c9d78881ac49fc WatchSource:0}: Error finding container 24e3edd31763bed9d120fd2bae76fdc17bf1b9570e4e836d96c9d78881ac49fc: Status 404 returned error can't find the container with id 24e3edd31763bed9d120fd2bae76fdc17bf1b9570e4e836d96c9d78881ac49fc Jul 10 20:42:22.972091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:22.972049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" event={"ID":"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5","Type":"ContainerStarted","Data":"32813b3c7bc5bbaa780b796b0fd642e90c3b21e20f45a8ca9265dd48e19ad207"} Jul 10 20:42:22.972585 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:22.972111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" event={"ID":"d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5","Type":"ContainerStarted","Data":"24e3edd31763bed9d120fd2bae76fdc17bf1b9570e4e836d96c9d78881ac49fc"} Jul 10 20:42:22.972585 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:22.972165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:22.990110 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:22.990059 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" podStartSLOduration=1.990043899 podStartE2EDuration="1.990043899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:22.989186793 +0000 UTC m=+273.595943387" watchObservedRunningTime="2026-07-10 20:42:22.990043899 +0000 UTC m=+273.596800535" Jul 10 20:42:30.867832 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:30.867779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:30.870242 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:30.870206 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fee6004-aff7-41a0-822b-bb6bae06d810-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2r288\" (UID: \"6fee6004-aff7-41a0-822b-bb6bae06d810\") " pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:30.880221 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:30.880196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2r288" Jul 10 20:42:31.004731 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:31.004698 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2r288"] Jul 10 20:42:31.007585 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:42:31.007558 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fee6004_aff7_41a0_822b_bb6bae06d810.slice/crio-ed3204e244e8a15664474d0c1f185cbafa8efcb4ac9c81c40391ca5839235678 WatchSource:0}: Error finding container ed3204e244e8a15664474d0c1f185cbafa8efcb4ac9c81c40391ca5839235678: Status 404 returned error can't find the container with id ed3204e244e8a15664474d0c1f185cbafa8efcb4ac9c81c40391ca5839235678 Jul 10 20:42:32.003323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:32.003223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2r288" event={"ID":"6fee6004-aff7-41a0-822b-bb6bae06d810","Type":"ContainerStarted","Data":"ed3204e244e8a15664474d0c1f185cbafa8efcb4ac9c81c40391ca5839235678"} Jul 10 20:42:33.650148 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:33.650088 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n66mb" podUID="ce5621c9-1df4-4e9d-87a1-335f497c90bc" Jul 10 20:42:34.013537 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:34.013505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:36.020962 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:36.020926 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2r288" event={"ID":"6fee6004-aff7-41a0-822b-bb6bae06d810","Type":"ContainerStarted","Data":"8adf3447be69dfe3da9213eb854d371dc66a3548179133f1f435bdd27f666edb"} Jul 10 20:42:36.021459 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:36.020996 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2r288" event={"ID":"6fee6004-aff7-41a0-822b-bb6bae06d810","Type":"ContainerStarted","Data":"89508313e1e6cb23d80f9e9db52bb72751d30d2a5a47f014d9da089a222e3aa1"} Jul 10 20:42:36.817456 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:36.817411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:36.820475 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:36.820443 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce5621c9-1df4-4e9d-87a1-335f497c90bc-metrics-tls\") pod \"dns-default-n66mb\" (UID: \"ce5621c9-1df4-4e9d-87a1-335f497c90bc\") " pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:37.016710 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.016675 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h798g\"" Jul 10 20:42:37.025275 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.025220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:37.972272 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.972207 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tddc"] Jul 10 20:42:37.987931 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.987902 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:37.990035 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.989993 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 20:42:37.990035 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990026 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 20:42:37.990278 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990047 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 20:42:37.990278 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990003 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-chckk\"" Jul 10 20:42:37.990278 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990003 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 20:42:37.990498 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990481 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 20:42:37.990543 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:37.990516 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 20:42:38.127809 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.127782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-sys\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.127825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mq9c\" (UniqueName: \"kubernetes.io/projected/d269469c-ec62-4c52-907d-bfc8256b81f6-kube-api-access-5mq9c\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.127899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-wtmp\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.127939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-textfile\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.128057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-metrics-client-ca\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.128090 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.128137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-tls\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128575 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.128194 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.128575 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.128222 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-root\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.135130 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.135034 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n66mb"] Jul 10 20:42:38.137592 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:42:38.137563 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5621c9_1df4_4e9d_87a1_335f497c90bc.slice/crio-ae3e78580086155d9b3194ab14584f285152c799fa1c4a43dfd187c8c7951292 WatchSource:0}: Error finding container ae3e78580086155d9b3194ab14584f285152c799fa1c4a43dfd187c8c7951292: Status 404 returned error can't find the container with id ae3e78580086155d9b3194ab14584f285152c799fa1c4a43dfd187c8c7951292 Jul 10 20:42:38.229031 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.228935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-wtmp\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229031 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.228976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-textfile\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229031 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-metrics-client-ca\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229078 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-tls\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-wtmp\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-root\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-root\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-sys\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mq9c\" (UniqueName: \"kubernetes.io/projected/d269469c-ec62-4c52-907d-bfc8256b81f6-kube-api-access-5mq9c\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229331 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229302 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d269469c-ec62-4c52-907d-bfc8256b81f6-sys\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229722 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-textfile\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229722 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.229802 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.229711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d269469c-ec62-4c52-907d-bfc8256b81f6-metrics-client-ca\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.231655 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.231634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.231748 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.231733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d269469c-ec62-4c52-907d-bfc8256b81f6-node-exporter-tls\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.238150 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.238123 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mq9c\" (UniqueName: \"kubernetes.io/projected/d269469c-ec62-4c52-907d-bfc8256b81f6-kube-api-access-5mq9c\") pod \"node-exporter-9tddc\" (UID: \"d269469c-ec62-4c52-907d-bfc8256b81f6\") " pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.298127 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:38.298097 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tddc" Jul 10 20:42:38.309263 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:42:38.309223 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd269469c_ec62_4c52_907d_bfc8256b81f6.slice/crio-51a0c6434b0072cec3d214b1227402dbbd10a468a133fffe38b2b8f671365cb9 WatchSource:0}: Error finding container 51a0c6434b0072cec3d214b1227402dbbd10a468a133fffe38b2b8f671365cb9: Status 404 returned error can't find the container with id 51a0c6434b0072cec3d214b1227402dbbd10a468a133fffe38b2b8f671365cb9 Jul 10 20:42:39.033008 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:39.032963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2r288" event={"ID":"6fee6004-aff7-41a0-822b-bb6bae06d810","Type":"ContainerStarted","Data":"e1b64b2cc11bed99285bc14ddf38b74860b83f7815e3bb41617092fc363b3ab2"} Jul 10 20:42:39.034370 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:39.034340 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tddc" event={"ID":"d269469c-ec62-4c52-907d-bfc8256b81f6","Type":"ContainerStarted","Data":"51a0c6434b0072cec3d214b1227402dbbd10a468a133fffe38b2b8f671365cb9"} Jul 10 20:42:39.035627 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:39.035596 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n66mb" event={"ID":"ce5621c9-1df4-4e9d-87a1-335f497c90bc","Type":"ContainerStarted","Data":"ae3e78580086155d9b3194ab14584f285152c799fa1c4a43dfd187c8c7951292"} Jul 10 20:42:39.049532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:39.049479 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2r288" podStartSLOduration=33.117681047 podStartE2EDuration="40.049461879s" totalImagesPullingTime="6.931780832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:41:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:39.04872382 +0000 UTC m=+289.655480431" watchObservedRunningTime="2026-07-10 20:42:39.049461879 +0000 UTC m=+289.656218462" Jul 10 20:42:40.040245 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.040191 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n66mb" event={"ID":"ce5621c9-1df4-4e9d-87a1-335f497c90bc","Type":"ContainerStarted","Data":"615c465a2806542b83442b00d4f69d7f157b57301fa0a66156087a815e32c76d"} Jul 10 20:42:40.040652 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.040272 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n66mb" event={"ID":"ce5621c9-1df4-4e9d-87a1-335f497c90bc","Type":"ContainerStarted","Data":"f3df8002f831c0584a9682e75b143039a15d8d4d58a2c7806f76931bc36e6dff"} Jul 10 20:42:40.040652 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.040300 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:40.041810 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.041786 2585 generic.go:356] "Generic (PLEG): container finished" podID="d269469c-ec62-4c52-907d-bfc8256b81f6" containerID="48c694dc7526b716fe3909a332c90ecc0b14769221d36138e3df4cd48ce20ee7" exitCode=0 Jul 10 20:42:40.041909 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.041847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tddc" event={"ID":"d269469c-ec62-4c52-907d-bfc8256b81f6","Type":"ContainerDied","Data":"48c694dc7526b716fe3909a332c90ecc0b14769221d36138e3df4cd48ce20ee7"} Jul 10 20:42:40.056448 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:40.056376 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n66mb" podStartSLOduration=252.350674657 podStartE2EDuration="4m14.056360898s" totalImagesPullingTime="1.705686241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:40.055543409 +0000 UTC m=+290.662300002" watchObservedRunningTime="2026-07-10 20:42:40.056360898 +0000 UTC m=+290.663117480" Jul 10 20:42:41.047135 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:41.047088 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tddc" event={"ID":"d269469c-ec62-4c52-907d-bfc8256b81f6","Type":"ContainerStarted","Data":"f473bc75374f2e377b3c6177c978eeda62162162c8f50aab6af1743ac3833e91"} Jul 10 20:42:41.047630 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:41.047162 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tddc" event={"ID":"d269469c-ec62-4c52-907d-bfc8256b81f6","Type":"ContainerStarted","Data":"7b437f293767a44f969b96e56a4c1192fdb97a635b7957caf26b74e68d8f6b28"} Jul 10 20:42:41.067142 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:41.067092 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tddc" podStartSLOduration=3.01541007 podStartE2EDuration="4.06707713s" totalImagesPullingTime="972.88961ms" totalInitContainerRuntime="78.77745ms" isStatefulPod=true podCreationTimestamp="2026-07-10 20:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:41.066391725 +0000 UTC m=+291.673148313" watchObservedRunningTime="2026-07-10 20:42:41.06707713 +0000 UTC m=+291.673833706" Jul 10 20:42:41.959948 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:41.959907 2585 patch_prober.go:28] interesting pod/image-registry-84fb596665-kl4sk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 20:42:41.960112 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:41.959967 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" podUID="d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 20:42:43.129022 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.128981 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fdb797dbd-p86t2"] Jul 10 20:42:43.132302 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.132279 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.134324 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134302 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 20:42:43.134324 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134319 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 20:42:43.134482 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134449 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 20:42:43.134675 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134661 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 20:42:43.134775 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134759 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j786w\"" Jul 10 20:42:43.134826 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.134762 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 20:42:43.139474 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.139187 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 20:42:43.144623 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.144601 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fdb797dbd-p86t2"] Jul 10 20:42:43.270303 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270303 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270306 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-metrics-client-ca\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270365 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-serving-certs-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm444\" (UniqueName: \"kubernetes.io/projected/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-kube-api-access-gm444\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-federate-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.270572 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.270556 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.371786 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.371786 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-metrics-client-ca\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-serving-certs-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm444\" (UniqueName: \"kubernetes.io/projected/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-kube-api-access-gm444\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-federate-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371929 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.371956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372599 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.372575 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-metrics-client-ca\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.372695 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.372639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-serving-certs-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.373170 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.373146 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.374771 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.374743 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-telemeter-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.374867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.374771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-federate-client-tls\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.374992 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.374974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.375047 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.375032 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-secret-telemeter-client\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.379908 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.379853 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm444\" (UniqueName: \"kubernetes.io/projected/bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8-kube-api-access-gm444\") pod \"telemeter-client-fdb797dbd-p86t2\" (UID: \"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8\") " pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.442499 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.442453 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" Jul 10 20:42:43.586130 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.586102 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fdb797dbd-p86t2"] Jul 10 20:42:43.587429 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:42:43.587401 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcabdaab_ba60_4aa7_a1f9_f49f54ddd0c8.slice/crio-0b97a97e0229886bcd1afd682593f3064c32dea7b0401b84d313e3339951fed7 WatchSource:0}: Error finding container 0b97a97e0229886bcd1afd682593f3064c32dea7b0401b84d313e3339951fed7: Status 404 returned error can't find the container with id 0b97a97e0229886bcd1afd682593f3064c32dea7b0401b84d313e3339951fed7 Jul 10 20:42:43.983756 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:43.983731 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84fb596665-kl4sk" Jul 10 20:42:44.057376 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:44.057336 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" event={"ID":"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8","Type":"ContainerStarted","Data":"0b97a97e0229886bcd1afd682593f3064c32dea7b0401b84d313e3339951fed7"} Jul 10 20:42:46.070253 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:46.070189 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" event={"ID":"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8","Type":"ContainerStarted","Data":"d1d8662657957a5c50f90df43b095986e802b12fe5580f2b154bbc9f27599f95"} Jul 10 20:42:47.075357 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:47.075322 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" event={"ID":"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8","Type":"ContainerStarted","Data":"04d847f84b4b4b017674bd93459a33a8b7bfa63028bc07fc71e10de6af3a5b99"} Jul 10 20:42:47.075749 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:47.075381 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" event={"ID":"bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8","Type":"ContainerStarted","Data":"76e0bac844ca2b82d0d29fa3ddb6f8d7503afbff2a0a06014b74375c477a22ae"} Jul 10 20:42:47.097012 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:47.096956 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fdb797dbd-p86t2" podStartSLOduration=1.166468393 podStartE2EDuration="4.096939699s" totalImagesPullingTime="2.930471306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:42:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:47.094864256 +0000 UTC m=+297.701620838" watchObservedRunningTime="2026-07-10 20:42:47.096939699 +0000 UTC m=+297.703696281" Jul 10 20:42:47.841933 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:47.841890 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" podUID="6442f444-608c-4596-8ba8-cadf0a894f37" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:42:47.980577 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:42:47.980533 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vnrsl" podUID="f525f08f-e462-48e9-8530-85773208a545" Jul 10 20:42:49.820635 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:49.820608 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:42:49.821087 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:49.820850 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:42:49.826637 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:49.826614 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 20:42:50.049574 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:50.049434 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n66mb" Jul 10 20:42:57.841847 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:57.841806 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" podUID="6442f444-608c-4596-8ba8-cadf0a894f37" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:42:58.979914 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:42:58.979869 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:43:07.841420 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:07.841379 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" podUID="6442f444-608c-4596-8ba8-cadf0a894f37" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:43:07.841801 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:07.841455 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" Jul 10 20:43:07.842021 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:07.841982 2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"192a42b457db27e1fbb8fb5d6b6d55b9e25edcffa7b1a17a0d23286c48ff3cdd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 20:43:07.842064 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:07.842037 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" podUID="6442f444-608c-4596-8ba8-cadf0a894f37" containerName="service-proxy" containerID="cri-o://192a42b457db27e1fbb8fb5d6b6d55b9e25edcffa7b1a17a0d23286c48ff3cdd" gracePeriod=30 Jul 10 20:43:08.149905 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:08.149819 2585 generic.go:356] "Generic (PLEG): container finished" podID="6442f444-608c-4596-8ba8-cadf0a894f37" containerID="192a42b457db27e1fbb8fb5d6b6d55b9e25edcffa7b1a17a0d23286c48ff3cdd" exitCode=2 Jul 10 20:43:08.149905 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:08.149887 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerDied","Data":"192a42b457db27e1fbb8fb5d6b6d55b9e25edcffa7b1a17a0d23286c48ff3cdd"} Jul 10 20:43:09.156598 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:43:09.156563 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bfccbd794-s4c65" event={"ID":"6442f444-608c-4596-8ba8-cadf0a894f37","Type":"ContainerStarted","Data":"6529adf8c50ecb5bf2484af24262cd957e9f6d42fac029087b317cd03854e8fb"} Jul 10 20:44:03.900687 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:03.900640 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:44:03.903074 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:03.903054 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f525f08f-e462-48e9-8530-85773208a545-metrics-certs\") pod \"network-metrics-daemon-vnrsl\" (UID: \"f525f08f-e462-48e9-8530-85773208a545\") " pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:44:04.082167 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:04.082131 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6zkh\"" Jul 10 20:44:04.090664 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:04.090640 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnrsl" Jul 10 20:44:04.214364 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:04.214194 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vnrsl"] Jul 10 20:44:04.217019 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:44:04.216992 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf525f08f_e462_48e9_8530_85773208a545.slice/crio-24f292bea11e0ed9f2c4be9d77424205f81fc52953feba288506c70dec4f92b1 WatchSource:0}: Error finding container 24f292bea11e0ed9f2c4be9d77424205f81fc52953feba288506c70dec4f92b1: Status 404 returned error can't find the container with id 24f292bea11e0ed9f2c4be9d77424205f81fc52953feba288506c70dec4f92b1 Jul 10 20:44:04.218866 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:04.218850 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:44:04.342997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:04.342953 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnrsl" event={"ID":"f525f08f-e462-48e9-8530-85773208a545","Type":"ContainerStarted","Data":"24f292bea11e0ed9f2c4be9d77424205f81fc52953feba288506c70dec4f92b1"} Jul 10 20:44:06.352214 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:06.352175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnrsl" event={"ID":"f525f08f-e462-48e9-8530-85773208a545","Type":"ContainerStarted","Data":"83a80cb5ff4bbc63edb9687363e9c552f0fa1f561499a436f32d443dfc44cdf1"} Jul 10 20:44:06.352629 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:06.352297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnrsl" event={"ID":"f525f08f-e462-48e9-8530-85773208a545","Type":"ContainerStarted","Data":"590bf8c3e7f3874791e239cb497e1525867a1c2a610fa4388a2055a40b69a2f9"} Jul 10 20:44:06.367550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:44:06.367491 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vnrsl" podStartSLOduration=375.151984492 podStartE2EDuration="6m16.367474129s" totalImagesPullingTime="1.215489637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:44:06.366365167 +0000 UTC m=+376.973121773" watchObservedRunningTime="2026-07-10 20:44:06.367474129 +0000 UTC m=+376.974230707" Jul 10 20:45:49.165174 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.165141 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px"] Jul 10 20:45:49.168133 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.168105 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.169896 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.169877 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-55xpd\"" Jul 10 20:45:49.169997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.169908 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 20:45:49.169997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.169978 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:45:49.178837 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.178815 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px"] Jul 10 20:45:49.268794 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.268765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrv7\" (UniqueName: \"kubernetes.io/projected/02d86451-732e-4ae7-ad02-08a619fc0de4-kube-api-access-2qrv7\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.268953 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.268855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02d86451-732e-4ae7-ad02-08a619fc0de4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.370086 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.370047 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02d86451-732e-4ae7-ad02-08a619fc0de4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.370321 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.370101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrv7\" (UniqueName: \"kubernetes.io/projected/02d86451-732e-4ae7-ad02-08a619fc0de4-kube-api-access-2qrv7\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.370521 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.370499 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02d86451-732e-4ae7-ad02-08a619fc0de4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.385558 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.385534 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrv7\" (UniqueName: \"kubernetes.io/projected/02d86451-732e-4ae7-ad02-08a619fc0de4-kube-api-access-2qrv7\") pod \"openshift-lws-operator-fd8ccff4c-x77px\" (UID: \"02d86451-732e-4ae7-ad02-08a619fc0de4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.477250 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.477198 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" Jul 10 20:45:49.604832 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.604808 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px"] Jul 10 20:45:49.607354 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:45:49.607325 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d86451_732e_4ae7_ad02_08a619fc0de4.slice/crio-aae428c8589d275371cb6cf1fff60b94b7786710e0e443fbc0d5528fe692dfe6 WatchSource:0}: Error finding container aae428c8589d275371cb6cf1fff60b94b7786710e0e443fbc0d5528fe692dfe6: Status 404 returned error can't find the container with id aae428c8589d275371cb6cf1fff60b94b7786710e0e443fbc0d5528fe692dfe6 Jul 10 20:45:49.692351 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:49.692316 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" event={"ID":"02d86451-732e-4ae7-ad02-08a619fc0de4","Type":"ContainerStarted","Data":"aae428c8589d275371cb6cf1fff60b94b7786710e0e443fbc0d5528fe692dfe6"} Jul 10 20:45:52.706736 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:45:52.706694 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" event={"ID":"02d86451-732e-4ae7-ad02-08a619fc0de4","Type":"ContainerStarted","Data":"c04a7c56bf45b7a6bd13e0cf86af6004d76380e061620dba01c4e3049ef3b6de"} Jul 10 20:46:09.003496 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.003433 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x77px" podStartSLOduration=17.54427093 podStartE2EDuration="20.003415152s" totalImagesPullingTime="2.459144222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:45:52.722899041 +0000 UTC m=+483.329655624" watchObservedRunningTime="2026-07-10 20:46:09.003415152 +0000 UTC m=+499.610171736" Jul 10 20:46:09.004081 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.004064 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x"] Jul 10 20:46:09.012400 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.012378 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.014183 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.014143 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 20:46:09.014365 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.014200 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 20:46:09.014424 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.014336 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 20:46:09.014545 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.014529 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 20:46:09.014625 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.014539 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cndqb\"" Jul 10 20:46:09.023798 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.023778 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x"] Jul 10 20:46:09.120294 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.120256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-tmp\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.120294 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.120297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ngn\" (UniqueName: \"kubernetes.io/projected/64818ea5-07a6-46fc-a9f9-a92f07a3d717-kube-api-access-f2ngn\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.120496 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.120332 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.120496 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.120379 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-manifests\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.120496 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.120415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-webhook-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-tmp\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ngn\" (UniqueName: \"kubernetes.io/projected/64818ea5-07a6-46fc-a9f9-a92f07a3d717-kube-api-access-f2ngn\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221577 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221577 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221386 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-manifests\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221577 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-webhook-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.221840 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.221819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-manifests\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.223976 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.223943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64818ea5-07a6-46fc-a9f9-a92f07a3d717-tmp\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.224070 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.223998 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-webhook-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.224070 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.224039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64818ea5-07a6-46fc-a9f9-a92f07a3d717-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.230552 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.230524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ngn\" (UniqueName: \"kubernetes.io/projected/64818ea5-07a6-46fc-a9f9-a92f07a3d717-kube-api-access-f2ngn\") pod \"opendatahub-operator-controller-manager-6558bffdbc-vsh7x\" (UID: \"64818ea5-07a6-46fc-a9f9-a92f07a3d717\") " pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.322613 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.322512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:09.471170 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.471092 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x"] Jul 10 20:46:09.474297 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:09.474269 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64818ea5_07a6_46fc_a9f9_a92f07a3d717.slice/crio-83b6aba17bc3a5dbdbff8c821e68463e90a8042f81634579f401ca6ecb234c52 WatchSource:0}: Error finding container 83b6aba17bc3a5dbdbff8c821e68463e90a8042f81634579f401ca6ecb234c52: Status 404 returned error can't find the container with id 83b6aba17bc3a5dbdbff8c821e68463e90a8042f81634579f401ca6ecb234c52 Jul 10 20:46:09.769672 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:09.769635 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" event={"ID":"64818ea5-07a6-46fc-a9f9-a92f07a3d717","Type":"ContainerStarted","Data":"83b6aba17bc3a5dbdbff8c821e68463e90a8042f81634579f401ca6ecb234c52"} Jul 10 20:46:12.782630 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:12.782593 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" event={"ID":"64818ea5-07a6-46fc-a9f9-a92f07a3d717","Type":"ContainerStarted","Data":"b430aa30e89f691cf0c4218ec9fc0950dc7b078935ff0c6914be630cb1460ff4"} Jul 10 20:46:13.788738 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:13.788654 2585 generic.go:356] "Generic (PLEG): container finished" podID="64818ea5-07a6-46fc-a9f9-a92f07a3d717" containerID="b430aa30e89f691cf0c4218ec9fc0950dc7b078935ff0c6914be630cb1460ff4" exitCode=0 Jul 10 20:46:13.788738 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:13.788703 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" event={"ID":"64818ea5-07a6-46fc-a9f9-a92f07a3d717","Type":"ContainerDied","Data":"b430aa30e89f691cf0c4218ec9fc0950dc7b078935ff0c6914be630cb1460ff4"} Jul 10 20:46:14.794808 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:14.794765 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" event={"ID":"64818ea5-07a6-46fc-a9f9-a92f07a3d717","Type":"ContainerStarted","Data":"49e92d76d2f0a0d004bb39f5e22140fa8e79829a00eee4c96420ac4bb27fd45f"} Jul 10 20:46:14.795210 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:14.794946 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:14.815857 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:14.815797 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" podStartSLOduration=2.358276902 podStartE2EDuration="6.815774741s" totalImagesPullingTime="3.383732021s" totalInitContainerRuntime="1.073765818s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:14.815001004 +0000 UTC m=+505.421757587" watchObservedRunningTime="2026-07-10 20:46:14.815774741 +0000 UTC m=+505.422531327" Jul 10 20:46:25.800910 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:25.800877 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6558bffdbc-vsh7x" Jul 10 20:46:29.240770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.240733 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq"] Jul 10 20:46:29.248409 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.248383 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.252644 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.252613 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 20:46:29.253625 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.253389 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 20:46:29.253625 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.253411 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 20:46:29.253625 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.253455 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 20:46:29.253879 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.253741 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dmhn5\"" Jul 10 20:46:29.260740 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.260711 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq"] Jul 10 20:46:29.388043 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.388006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa0555-e386-4867-ae3b-00bec97eb705-tmp\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.388224 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.388059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aa0555-e386-4867-ae3b-00bec97eb705-tls-certs\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.388224 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.388137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6g7\" (UniqueName: \"kubernetes.io/projected/90aa0555-e386-4867-ae3b-00bec97eb705-kube-api-access-zt6g7\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.439673 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.439635 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xhdhj"] Jul 10 20:46:29.442697 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.442676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.445141 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:29.445108 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-xhdhj\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="cc8a48f1-5618-4d1c-b298-00a7ec08f796" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.445302 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:29.445167 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-p5f8f\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p5f8f\"" type="*v1.Secret" Jul 10 20:46:29.445302 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:29.445217 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" Jul 10 20:46:29.466374 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.466342 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xhdhj"] Jul 10 20:46:29.489283 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.489255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aa0555-e386-4867-ae3b-00bec97eb705-tls-certs\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.489463 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.489298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6g7\" (UniqueName: \"kubernetes.io/projected/90aa0555-e386-4867-ae3b-00bec97eb705-kube-api-access-zt6g7\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.489463 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.489358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa0555-e386-4867-ae3b-00bec97eb705-tmp\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.491942 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.491874 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa0555-e386-4867-ae3b-00bec97eb705-tmp\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.492134 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.492112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aa0555-e386-4867-ae3b-00bec97eb705-tls-certs\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.512440 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.512402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6g7\" (UniqueName: \"kubernetes.io/projected/90aa0555-e386-4867-ae3b-00bec97eb705-kube-api-access-zt6g7\") pod \"kube-auth-proxy-86d9dbc65b-nm2qq\" (UID: \"90aa0555-e386-4867-ae3b-00bec97eb705\") " pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.559280 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.559245 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" Jul 10 20:46:29.590645 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.590610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc8a48f1-5618-4d1c-b298-00a7ec08f796-cert\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.590777 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.590707 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crndc\" (UniqueName: \"kubernetes.io/projected/cc8a48f1-5618-4d1c-b298-00a7ec08f796-kube-api-access-crndc\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.691750 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.691718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crndc\" (UniqueName: \"kubernetes.io/projected/cc8a48f1-5618-4d1c-b298-00a7ec08f796-kube-api-access-crndc\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.691908 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.691797 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc8a48f1-5618-4d1c-b298-00a7ec08f796-cert\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.698756 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.698721 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq"] Jul 10 20:46:29.702055 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.702030 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crndc\" (UniqueName: \"kubernetes.io/projected/cc8a48f1-5618-4d1c-b298-00a7ec08f796-kube-api-access-crndc\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:29.702191 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:29.702121 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aa0555_e386_4867_ae3b_00bec97eb705.slice/crio-d509111e19559ec7a2f00266f11659b1ee491f996a8c6c427a04b43076f40c9a WatchSource:0}: Error finding container d509111e19559ec7a2f00266f11659b1ee491f996a8c6c427a04b43076f40c9a: Status 404 returned error can't find the container with id d509111e19559ec7a2f00266f11659b1ee491f996a8c6c427a04b43076f40c9a Jul 10 20:46:29.848062 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:29.847975 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" event={"ID":"90aa0555-e386-4867-ae3b-00bec97eb705","Type":"ContainerStarted","Data":"d509111e19559ec7a2f00266f11659b1ee491f996a8c6c427a04b43076f40c9a"} Jul 10 20:46:30.361143 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.361112 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgpmz"] Jul 10 20:46:30.365365 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.365343 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.367091 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.367071 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 20:46:30.372014 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.371992 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 20:46:30.373960 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.373935 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgpmz"] Jul 10 20:46:30.384323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.384298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc8a48f1-5618-4d1c-b298-00a7ec08f796-cert\") pod \"odh-model-controller-7998c67dc4-xhdhj\" (UID: \"cc8a48f1-5618-4d1c-b298-00a7ec08f796\") " pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:30.404005 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.403972 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:30.407388 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.407364 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.409640 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.409609 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 20:46:30.409739 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.409609 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 20:46:30.409867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.409852 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 20:46:30.409939 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.409877 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nfb7v\"" Jul 10 20:46:30.409939 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.409883 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 20:46:30.431728 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.431696 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:30.498973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.498934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23a37d2-5644-4d39-bb48-eb7ddb047705-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.498987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499037 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499062 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzhc\" (UniqueName: \"kubernetes.io/projected/e23a37d2-5644-4d39-bb48-eb7ddb047705-kube-api-access-4zzhc\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499131 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b52m\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499185 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499599 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499199 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.499599 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.499265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.599787 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23a37d2-5644-4d39-bb48-eb7ddb047705-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzhc\" (UniqueName: \"kubernetes.io/projected/e23a37d2-5644-4d39-bb48-eb7ddb047705-kube-api-access-4zzhc\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599955 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.599981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b52m\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.600018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600039 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.600050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.600737 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.600530 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.603545 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.603496 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.604067 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.603999 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.604067 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.604022 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.604214 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.604153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.604214 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.604162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23a37d2-5644-4d39-bb48-eb7ddb047705-tls-certs\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.608188 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.608164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzhc\" (UniqueName: \"kubernetes.io/projected/e23a37d2-5644-4d39-bb48-eb7ddb047705-kube-api-access-4zzhc\") pod \"model-serving-api-54d89bd5fc-mgpmz\" (UID: \"e23a37d2-5644-4d39-bb48-eb7ddb047705\") " pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.609098 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.609069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.609366 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.609343 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b52m\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m\") pod \"istiod-openshift-gateway-7574459656-ndvsz\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.676848 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.676758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:30.720257 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.718990 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:30.844393 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.844127 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mgpmz"] Jul 10 20:46:30.847908 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:30.847876 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23a37d2_5644_4d39_bb48_eb7ddb047705.slice/crio-16b142ef896980284fd02d10f5c7191375ab2e71c4437fb27526fdfae975b1c8 WatchSource:0}: Error finding container 16b142ef896980284fd02d10f5c7191375ab2e71c4437fb27526fdfae975b1c8: Status 404 returned error can't find the container with id 16b142ef896980284fd02d10f5c7191375ab2e71c4437fb27526fdfae975b1c8 Jul 10 20:46:30.861345 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.861299 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" event={"ID":"e23a37d2-5644-4d39-bb48-eb7ddb047705","Type":"ContainerStarted","Data":"16b142ef896980284fd02d10f5c7191375ab2e71c4437fb27526fdfae975b1c8"} Jul 10 20:46:30.892909 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:30.892878 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:30.895040 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:30.895006 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fdfb8_178c_4d6a_a2ba_89a806ebe36f.slice/crio-341dc9f9c696c806475b956163d1c029793d5b12e0c41f8ea119892c52a9a9b4 WatchSource:0}: Error finding container 341dc9f9c696c806475b956163d1c029793d5b12e0c41f8ea119892c52a9a9b4: Status 404 returned error can't find the container with id 341dc9f9c696c806475b956163d1c029793d5b12e0c41f8ea119892c52a9a9b4 Jul 10 20:46:31.008302 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:31.008268 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p5f8f\"" Jul 10 20:46:31.012705 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:31.012687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:31.170611 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:31.170569 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xhdhj"] Jul 10 20:46:31.173995 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:31.173962 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8a48f1_5618_4d1c_b298_00a7ec08f796.slice/crio-d1061820da39ddd280ded50bb3c715f0d1bee7cc6f14537ddbfd61e543661160 WatchSource:0}: Error finding container d1061820da39ddd280ded50bb3c715f0d1bee7cc6f14537ddbfd61e543661160: Status 404 returned error can't find the container with id d1061820da39ddd280ded50bb3c715f0d1bee7cc6f14537ddbfd61e543661160 Jul 10 20:46:31.868085 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:31.868041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" event={"ID":"cc8a48f1-5618-4d1c-b298-00a7ec08f796","Type":"ContainerStarted","Data":"d1061820da39ddd280ded50bb3c715f0d1bee7cc6f14537ddbfd61e543661160"} Jul 10 20:46:31.869471 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:31.869406 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" event={"ID":"251fdfb8-178c-4d6a-a2ba-89a806ebe36f","Type":"ContainerStarted","Data":"341dc9f9c696c806475b956163d1c029793d5b12e0c41f8ea119892c52a9a9b4"} Jul 10 20:46:34.174378 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.174341 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 10 20:46:34.174770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.174421 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 10 20:46:34.886741 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.886703 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" event={"ID":"e23a37d2-5644-4d39-bb48-eb7ddb047705","Type":"ContainerStarted","Data":"9489f792f87f1f4bd414562758a8ae2a9bb469b115d35a867d3eb61377f34038"} Jul 10 20:46:34.886911 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.886823 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:34.888675 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.888638 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" event={"ID":"90aa0555-e386-4867-ae3b-00bec97eb705","Type":"ContainerStarted","Data":"0c7a7c4af3372246130982fc2f51f06b170d08420263ab62f7fd4724a93fe45b"} Jul 10 20:46:34.890636 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.890611 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" event={"ID":"251fdfb8-178c-4d6a-a2ba-89a806ebe36f","Type":"ContainerStarted","Data":"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3"} Jul 10 20:46:34.890767 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.890747 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:34.919040 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.918918 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86d9dbc65b-nm2qq" podStartSLOduration=1.455255427 podStartE2EDuration="5.918893473s" totalImagesPullingTime="4.463638046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:34.916656534 +0000 UTC m=+525.523413134" watchObservedRunningTime="2026-07-10 20:46:34.918893473 +0000 UTC m=+525.525650095" Jul 10 20:46:34.919776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.919734 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" podStartSLOduration=1.6021619070000002 podStartE2EDuration="4.919721319s" totalImagesPullingTime="3.317559412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:34.902739943 +0000 UTC m=+525.509496525" watchObservedRunningTime="2026-07-10 20:46:34.919721319 +0000 UTC m=+525.526477895" Jul 10 20:46:34.935528 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:34.935286 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" podStartSLOduration=1.6588074929999999 podStartE2EDuration="4.935261637s" totalImagesPullingTime="3.276454144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:34.932641202 +0000 UTC m=+525.539397810" watchObservedRunningTime="2026-07-10 20:46:34.935261637 +0000 UTC m=+525.542018222" Jul 10 20:46:35.897416 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:35.897382 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:36.902820 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:36.902765 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" event={"ID":"cc8a48f1-5618-4d1c-b298-00a7ec08f796","Type":"ContainerStarted","Data":"d9b284df1367952de541f702aa449464f2e657ff53b3f0ebf3044c29b91b0b7c"} Jul 10 20:46:36.903360 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:36.902880 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:46:36.918377 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:36.918314 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" podStartSLOduration=2.7269209930000002 podStartE2EDuration="7.918283663s" totalImagesPullingTime="5.19136267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:36.917622833 +0000 UTC m=+527.524379443" watchObservedRunningTime="2026-07-10 20:46:36.918283663 +0000 UTC m=+527.525040245" Jul 10 20:46:41.232867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.232787 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gtd22"] Jul 10 20:46:41.235955 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.235936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.238603 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.238585 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xtkb2\"" Jul 10 20:46:41.238701 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.238585 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 20:46:41.248872 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.248851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td"] Jul 10 20:46:41.251665 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.251646 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gtd22"] Jul 10 20:46:41.251761 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.251730 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.253656 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.253635 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 20:46:41.253757 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.253663 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dvzfc\"" Jul 10 20:46:41.269326 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.269284 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td"] Jul 10 20:46:41.402603 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.402550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/e2bceca8-b4f5-4902-83f4-07b9b062f51b-kube-api-access-wkqsb\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.402603 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.402616 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ccc476-bce0-4524-a89b-821a1e924a54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.402876 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.402721 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6zq\" (UniqueName: \"kubernetes.io/projected/b8ccc476-bce0-4524-a89b-821a1e924a54-kube-api-access-xm6zq\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.402876 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.402766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.503713 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.503587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6zq\" (UniqueName: \"kubernetes.io/projected/b8ccc476-bce0-4524-a89b-821a1e924a54-kube-api-access-xm6zq\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.503713 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.503646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.503713 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.503711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/e2bceca8-b4f5-4902-83f4-07b9b062f51b-kube-api-access-wkqsb\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.504055 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.503752 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ccc476-bce0-4524-a89b-821a1e924a54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.504055 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:41.503847 2585 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 20:46:41.504055 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:41.503919 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert podName:e2bceca8-b4f5-4902-83f4-07b9b062f51b nodeName:}" failed. No retries permitted until 2026-07-10 20:46:42.003897739 +0000 UTC m=+532.610654303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert") pod "kserve-controller-manager-d6f4b65b8-gtd22" (UID: "e2bceca8-b4f5-4902-83f4-07b9b062f51b") : secret "kserve-webhook-server-cert" not found Jul 10 20:46:41.506540 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.506516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8ccc476-bce0-4524-a89b-821a1e924a54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.519928 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.519895 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/e2bceca8-b4f5-4902-83f4-07b9b062f51b-kube-api-access-wkqsb\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:41.520120 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.520102 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6zq\" (UniqueName: \"kubernetes.io/projected/b8ccc476-bce0-4524-a89b-821a1e924a54-kube-api-access-xm6zq\") pod \"llmisvc-controller-manager-777c6bd7d5-k84td\" (UID: \"b8ccc476-bce0-4524-a89b-821a1e924a54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.561344 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.561312 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:41.703022 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.702996 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td"] Jul 10 20:46:41.705742 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:41.705714 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8ccc476_bce0_4524_a89b_821a1e924a54.slice/crio-f3a602a9ba8b3affd915be08bce0b8b7eaf19ff61509a87dca09793937cfba66 WatchSource:0}: Error finding container f3a602a9ba8b3affd915be08bce0b8b7eaf19ff61509a87dca09793937cfba66: Status 404 returned error can't find the container with id f3a602a9ba8b3affd915be08bce0b8b7eaf19ff61509a87dca09793937cfba66 Jul 10 20:46:41.929722 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:41.929630 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" event={"ID":"b8ccc476-bce0-4524-a89b-821a1e924a54","Type":"ContainerStarted","Data":"f3a602a9ba8b3affd915be08bce0b8b7eaf19ff61509a87dca09793937cfba66"} Jul 10 20:46:42.008083 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:42.008052 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:42.010666 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:42.010645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bceca8-b4f5-4902-83f4-07b9b062f51b-cert\") pod \"kserve-controller-manager-d6f4b65b8-gtd22\" (UID: \"e2bceca8-b4f5-4902-83f4-07b9b062f51b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:42.147100 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:42.147067 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:42.295084 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:42.295061 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gtd22"] Jul 10 20:46:42.296380 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:46:42.296358 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bceca8_b4f5_4902_83f4_07b9b062f51b.slice/crio-dcacc78bd8d85e02dd2702b998a9039ccdef4ce438fc0ee486f660208daa2051 WatchSource:0}: Error finding container dcacc78bd8d85e02dd2702b998a9039ccdef4ce438fc0ee486f660208daa2051: Status 404 returned error can't find the container with id dcacc78bd8d85e02dd2702b998a9039ccdef4ce438fc0ee486f660208daa2051 Jul 10 20:46:42.939026 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:42.938992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" event={"ID":"e2bceca8-b4f5-4902-83f4-07b9b062f51b","Type":"ContainerStarted","Data":"dcacc78bd8d85e02dd2702b998a9039ccdef4ce438fc0ee486f660208daa2051"} Jul 10 20:46:43.307596 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.307507 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:43.308050 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.307729 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" podUID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" containerName="discovery" containerID="cri-o://7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3" gracePeriod=30 Jul 10 20:46:43.592482 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.592457 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:43.723976 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.723943 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724003 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724030 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724093 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b52m\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724428 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724170 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs\") pod \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\" (UID: \"251fdfb8-178c-4d6a-a2ba-89a806ebe36f\") " Jul 10 20:46:43.724770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.724736 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:46:43.727126 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.727070 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts" (OuterVolumeSpecName: "cacerts") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:46:43.727579 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.727544 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:46:43.727579 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.727557 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token" (OuterVolumeSpecName: "istio-token") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:46:43.727895 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.727832 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m" (OuterVolumeSpecName: "kube-api-access-8b52m") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "kube-api-access-8b52m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:46:43.728518 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.728494 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs" (OuterVolumeSpecName: "local-certs") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:46:43.729551 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.729524 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "251fdfb8-178c-4d6a-a2ba-89a806ebe36f" (UID: "251fdfb8-178c-4d6a-a2ba-89a806ebe36f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.825956 2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-kubeconfig\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.825991 2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-cacerts\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.826001 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-dns-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.826012 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-csr-ca-configmap\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.826022 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b52m\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-kube-api-access-8b52m\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.826031 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-istio-token\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.826048 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.826039 2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdfb8-178c-4d6a-a2ba-89a806ebe36f-local-certs\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:46:43.943887 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.943852 2585 generic.go:356] "Generic (PLEG): container finished" podID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" containerID="7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3" exitCode=0 Jul 10 20:46:43.944063 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.943899 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" Jul 10 20:46:43.944063 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.943920 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" event={"ID":"251fdfb8-178c-4d6a-a2ba-89a806ebe36f","Type":"ContainerDied","Data":"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3"} Jul 10 20:46:43.944063 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.943960 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz" event={"ID":"251fdfb8-178c-4d6a-a2ba-89a806ebe36f","Type":"ContainerDied","Data":"341dc9f9c696c806475b956163d1c029793d5b12e0c41f8ea119892c52a9a9b4"} Jul 10 20:46:43.944063 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.943981 2585 scope.go:117] "RemoveContainer" containerID="7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3" Jul 10 20:46:43.952629 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.952613 2585 scope.go:117] "RemoveContainer" containerID="7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3" Jul 10 20:46:43.952896 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:46:43.952874 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3\": container with ID starting with 7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3 not found: ID does not exist" containerID="7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3" Jul 10 20:46:43.952944 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.952905 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3"} err="failed to get container status \"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3\": rpc error: code = NotFound desc = could not find container \"7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3\": container with ID starting with 7da553fa938786e74b28dced805f144a69e450a97718fd35296a07bf892e4df3 not found: ID does not exist" Jul 10 20:46:43.963835 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.963813 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:43.968040 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.968014 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ndvsz"] Jul 10 20:46:43.986417 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:43.986388 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" path="/var/lib/kubelet/pods/251fdfb8-178c-4d6a-a2ba-89a806ebe36f/volumes" Jul 10 20:46:45.900004 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.899973 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mgpmz" Jul 10 20:46:45.957946 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.957914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" event={"ID":"b8ccc476-bce0-4524-a89b-821a1e924a54","Type":"ContainerStarted","Data":"0c18fef51e3dfd4f7184aaca0837ada23263483c8f6c638092ce487bcb135088"} Jul 10 20:46:45.958127 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.958014 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:46:45.959770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.959744 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" event={"ID":"e2bceca8-b4f5-4902-83f4-07b9b062f51b","Type":"ContainerStarted","Data":"2d6e9e94d528d1284a67750aaae72ef6508330c3bb48e50c2a33b06369f717b9"} Jul 10 20:46:45.959939 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.959919 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:46:45.973152 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.973105 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" podStartSLOduration=1.474052188 podStartE2EDuration="4.973085208s" totalImagesPullingTime="3.49903302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:45.971772946 +0000 UTC m=+536.578529528" watchObservedRunningTime="2026-07-10 20:46:45.973085208 +0000 UTC m=+536.579841789" Jul 10 20:46:45.986770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:45.986724 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" podStartSLOduration=2.076684736 podStartE2EDuration="4.986709135s" totalImagesPullingTime="2.910024399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:45.984872375 +0000 UTC m=+536.591628978" watchObservedRunningTime="2026-07-10 20:46:45.986709135 +0000 UTC m=+536.593465764" Jul 10 20:46:47.909805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:46:47.909773 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xhdhj" Jul 10 20:47:16.966755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:16.966723 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-k84td" Jul 10 20:47:16.969866 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:16.969841 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gtd22" Jul 10 20:47:30.250898 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.250859 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:30.251299 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.251139 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" containerName="discovery" Jul 10 20:47:30.251299 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.251149 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" containerName="discovery" Jul 10 20:47:30.251299 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.251226 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="251fdfb8-178c-4d6a-a2ba-89a806ebe36f" containerName="discovery" Jul 10 20:47:30.255328 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.255307 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:30.257862 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.257842 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 20:47:30.257973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.257844 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dn6xt\"" Jul 10 20:47:30.258298 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.258284 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 20:47:30.263863 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.263758 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:30.406176 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.406145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62vf\" (UniqueName: \"kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf\") pod \"kuadrant-operator-catalog-7ccq2\" (UID: \"86cb85be-2891-42e4-b72e-59d8d2cef6a4\") " pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:30.507635 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.507542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62vf\" (UniqueName: \"kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf\") pod \"kuadrant-operator-catalog-7ccq2\" (UID: \"86cb85be-2891-42e4-b72e-59d8d2cef6a4\") " pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:30.518447 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.518415 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62vf\" (UniqueName: \"kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf\") pod \"kuadrant-operator-catalog-7ccq2\" (UID: \"86cb85be-2891-42e4-b72e-59d8d2cef6a4\") " pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:30.567515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.567471 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:30.696541 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.696510 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:30.698999 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:47:30.698975 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cb85be_2891_42e4_b72e_59d8d2cef6a4.slice/crio-b11e33c09ab5a6e8baa79978546c44210073b84f843b11ee2fb0c432a1418757 WatchSource:0}: Error finding container b11e33c09ab5a6e8baa79978546c44210073b84f843b11ee2fb0c432a1418757: Status 404 returned error can't find the container with id b11e33c09ab5a6e8baa79978546c44210073b84f843b11ee2fb0c432a1418757 Jul 10 20:47:30.732512 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.732478 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:30.824163 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.824081 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kdf75"] Jul 10 20:47:30.827352 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.827334 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:30.833986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:30.833955 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kdf75"] Jul 10 20:47:31.012023 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.011981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqfl\" (UniqueName: \"kubernetes.io/projected/ddf41688-34ed-40e0-8ceb-50f9c32cb95e-kube-api-access-zcqfl\") pod \"kuadrant-operator-catalog-kdf75\" (UID: \"ddf41688-34ed-40e0-8ceb-50f9c32cb95e\") " pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:31.113269 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.113154 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqfl\" (UniqueName: \"kubernetes.io/projected/ddf41688-34ed-40e0-8ceb-50f9c32cb95e-kube-api-access-zcqfl\") pod \"kuadrant-operator-catalog-kdf75\" (UID: \"ddf41688-34ed-40e0-8ceb-50f9c32cb95e\") " pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:31.120468 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.120436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqfl\" (UniqueName: \"kubernetes.io/projected/ddf41688-34ed-40e0-8ceb-50f9c32cb95e-kube-api-access-zcqfl\") pod \"kuadrant-operator-catalog-kdf75\" (UID: \"ddf41688-34ed-40e0-8ceb-50f9c32cb95e\") " pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:31.137026 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.136984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:31.138302 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.138276 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" event={"ID":"86cb85be-2891-42e4-b72e-59d8d2cef6a4","Type":"ContainerStarted","Data":"b11e33c09ab5a6e8baa79978546c44210073b84f843b11ee2fb0c432a1418757"} Jul 10 20:47:31.282133 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:31.282111 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kdf75"] Jul 10 20:47:31.284033 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:47:31.284002 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf41688_34ed_40e0_8ceb_50f9c32cb95e.slice/crio-1d775f30f1340bf968aefc7defe902abefae014225d7ce42543e7a29db747d3c WatchSource:0}: Error finding container 1d775f30f1340bf968aefc7defe902abefae014225d7ce42543e7a29db747d3c: Status 404 returned error can't find the container with id 1d775f30f1340bf968aefc7defe902abefae014225d7ce42543e7a29db747d3c Jul 10 20:47:32.145393 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:32.145315 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" event={"ID":"ddf41688-34ed-40e0-8ceb-50f9c32cb95e","Type":"ContainerStarted","Data":"1d775f30f1340bf968aefc7defe902abefae014225d7ce42543e7a29db747d3c"} Jul 10 20:47:34.155047 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.155010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" event={"ID":"ddf41688-34ed-40e0-8ceb-50f9c32cb95e","Type":"ContainerStarted","Data":"7034e9cf6167ca63a72e2f4bde4d3dcb953780616662bed74725ad7a8e5e2da0"} Jul 10 20:47:34.156645 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.156611 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" event={"ID":"86cb85be-2891-42e4-b72e-59d8d2cef6a4","Type":"ContainerStarted","Data":"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390"} Jul 10 20:47:34.156772 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.156662 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" podUID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" containerName="registry-server" containerID="cri-o://baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390" gracePeriod=2 Jul 10 20:47:34.168995 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.168945 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" podStartSLOduration=2.197302035 podStartE2EDuration="4.168929533s" totalImagesPullingTime="1.971627498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:47:34.168296482 +0000 UTC m=+584.775053085" watchObservedRunningTime="2026-07-10 20:47:34.168929533 +0000 UTC m=+584.775686114" Jul 10 20:47:34.182637 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.182586 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" podStartSLOduration=1.6298709470000001 podStartE2EDuration="4.182568818s" totalImagesPullingTime="2.552697871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:47:34.180719454 +0000 UTC m=+584.787476036" watchObservedRunningTime="2026-07-10 20:47:34.182568818 +0000 UTC m=+584.789325401" Jul 10 20:47:34.396839 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.396814 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:34.547269 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.547220 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j62vf\" (UniqueName: \"kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf\") pod \"86cb85be-2891-42e4-b72e-59d8d2cef6a4\" (UID: \"86cb85be-2891-42e4-b72e-59d8d2cef6a4\") " Jul 10 20:47:34.549752 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.549715 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf" (OuterVolumeSpecName: "kube-api-access-j62vf") pod "86cb85be-2891-42e4-b72e-59d8d2cef6a4" (UID: "86cb85be-2891-42e4-b72e-59d8d2cef6a4"). InnerVolumeSpecName "kube-api-access-j62vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:47:34.648398 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:34.648363 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j62vf\" (UniqueName: \"kubernetes.io/projected/86cb85be-2891-42e4-b72e-59d8d2cef6a4-kube-api-access-j62vf\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:47:35.162026 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.161991 2585 generic.go:356] "Generic (PLEG): container finished" podID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" containerID="baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390" exitCode=0 Jul 10 20:47:35.162443 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.162037 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" Jul 10 20:47:35.162443 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.162084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" event={"ID":"86cb85be-2891-42e4-b72e-59d8d2cef6a4","Type":"ContainerDied","Data":"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390"} Jul 10 20:47:35.162443 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.162132 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7ccq2" event={"ID":"86cb85be-2891-42e4-b72e-59d8d2cef6a4","Type":"ContainerDied","Data":"b11e33c09ab5a6e8baa79978546c44210073b84f843b11ee2fb0c432a1418757"} Jul 10 20:47:35.162443 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.162156 2585 scope.go:117] "RemoveContainer" containerID="baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390" Jul 10 20:47:35.171086 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.171066 2585 scope.go:117] "RemoveContainer" containerID="baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390" Jul 10 20:47:35.171387 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:47:35.171368 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390\": container with ID starting with baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390 not found: ID does not exist" containerID="baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390" Jul 10 20:47:35.171450 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.171398 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390"} err="failed to get container status \"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390\": rpc error: code = NotFound desc = could not find container \"baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390\": container with ID starting with baece1de9ef52f69e77fb3b1b709a8fba913da5053520c63bf4294a6966fe390 not found: ID does not exist" Jul 10 20:47:35.181588 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.181561 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:35.185859 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.185836 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7ccq2"] Jul 10 20:47:35.984461 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:35.984417 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" path="/var/lib/kubelet/pods/86cb85be-2891-42e4-b72e-59d8d2cef6a4/volumes" Jul 10 20:47:41.138142 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:41.138105 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:41.138632 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:41.138165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:41.161440 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:41.161412 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:41.209903 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:41.209878 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kdf75" Jul 10 20:47:49.850649 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:49.850618 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:47:49.852620 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:47:49.852599 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:48:02.679505 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.679468 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82"] Jul 10 20:48:02.687650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.679768 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" containerName="registry-server" Jul 10 20:48:02.687650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.679780 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" containerName="registry-server" Jul 10 20:48:02.687650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.679834 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cb85be-2891-42e4-b72e-59d8d2cef6a4" containerName="registry-server" Jul 10 20:48:02.687794 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.687657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:02.689728 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.689700 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hj2g2\"" Jul 10 20:48:02.693988 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.693962 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82"] Jul 10 20:48:02.773333 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.773297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h858h\" (UniqueName: \"kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h\") pod \"limitador-operator-controller-manager-859d7b78bd-fxs82\" (UID: \"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:02.874217 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.874179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h858h\" (UniqueName: \"kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h\") pod \"limitador-operator-controller-manager-859d7b78bd-fxs82\" (UID: \"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:02.883379 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.883346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h858h\" (UniqueName: \"kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h\") pod \"limitador-operator-controller-manager-859d7b78bd-fxs82\" (UID: \"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:02.998832 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:02.998789 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:03.137060 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:03.137037 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82"] Jul 10 20:48:03.139756 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:03.139728 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd291ebb_fdc3_452c_adcc_aa9b9afd4bea.slice/crio-b4b5d3b7ad019c0cc83add24a5b48f8288697b751cb3187e90259237f5a1140d WatchSource:0}: Error finding container b4b5d3b7ad019c0cc83add24a5b48f8288697b751cb3187e90259237f5a1140d: Status 404 returned error can't find the container with id b4b5d3b7ad019c0cc83add24a5b48f8288697b751cb3187e90259237f5a1140d Jul 10 20:48:03.278957 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:03.278856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" event={"ID":"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea","Type":"ContainerStarted","Data":"b4b5d3b7ad019c0cc83add24a5b48f8288697b751cb3187e90259237f5a1140d"} Jul 10 20:48:05.290498 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:05.290468 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" event={"ID":"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea","Type":"ContainerStarted","Data":"0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc"} Jul 10 20:48:05.290953 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:05.290560 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:05.305406 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:05.305351 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" podStartSLOduration=1.545932359 podStartE2EDuration="3.305334489s" totalImagesPullingTime="1.75940213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:05.303760094 +0000 UTC m=+615.910516682" watchObservedRunningTime="2026-07-10 20:48:05.305334489 +0000 UTC m=+615.912091070" Jul 10 20:48:06.129969 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.129933 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb"] Jul 10 20:48:06.133300 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.133280 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:06.135390 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.135367 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 20:48:06.135498 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.135415 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vpfnr\"" Jul 10 20:48:06.143730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.143706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb"] Jul 10 20:48:06.203671 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.203628 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/f1719f05-e730-4a7c-a11b-2656a92be333-kube-api-access-q4c5b\") pod \"dns-operator-controller-manager-864dd78c4c-njzfb\" (UID: \"f1719f05-e730-4a7c-a11b-2656a92be333\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:06.304525 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.304484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/f1719f05-e730-4a7c-a11b-2656a92be333-kube-api-access-q4c5b\") pod \"dns-operator-controller-manager-864dd78c4c-njzfb\" (UID: \"f1719f05-e730-4a7c-a11b-2656a92be333\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:06.315029 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.314990 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/f1719f05-e730-4a7c-a11b-2656a92be333-kube-api-access-q4c5b\") pod \"dns-operator-controller-manager-864dd78c4c-njzfb\" (UID: \"f1719f05-e730-4a7c-a11b-2656a92be333\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:06.443797 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.443753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:06.580759 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:06.580733 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb"] Jul 10 20:48:06.583338 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:06.583310 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1719f05_e730_4a7c_a11b_2656a92be333.slice/crio-2949cd4d7b6d12200d133664b08eeca80a2fbb3244abdc9f142e6467ae504505 WatchSource:0}: Error finding container 2949cd4d7b6d12200d133664b08eeca80a2fbb3244abdc9f142e6467ae504505: Status 404 returned error can't find the container with id 2949cd4d7b6d12200d133664b08eeca80a2fbb3244abdc9f142e6467ae504505 Jul 10 20:48:07.299608 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:07.299570 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" event={"ID":"f1719f05-e730-4a7c-a11b-2656a92be333","Type":"ContainerStarted","Data":"2949cd4d7b6d12200d133664b08eeca80a2fbb3244abdc9f142e6467ae504505"} Jul 10 20:48:09.308926 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:09.308891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" event={"ID":"f1719f05-e730-4a7c-a11b-2656a92be333","Type":"ContainerStarted","Data":"e8310b77987cd2fae5c1d2055690c204fb86523ce0a6c1797682c688d3c5bb77"} Jul 10 20:48:09.309328 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:09.309020 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:09.324199 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:09.324150 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" podStartSLOduration=1.026821539 podStartE2EDuration="3.324131637s" totalImagesPullingTime="2.297310098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:09.323444076 +0000 UTC m=+619.930200675" watchObservedRunningTime="2026-07-10 20:48:09.324131637 +0000 UTC m=+619.930888219" Jul 10 20:48:11.148130 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.148098 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t"] Jul 10 20:48:11.157804 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.157778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.160802 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.160775 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j6t8z\"" Jul 10 20:48:11.173148 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.173121 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t"] Jul 10 20:48:11.258843 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.258806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.258843 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.258843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27rl\" (UniqueName: \"kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.359288 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.359221 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.359288 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.359288 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27rl\" (UniqueName: \"kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.359688 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.359665 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.367618 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.367586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27rl\" (UniqueName: \"kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.469305 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.469269 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:11.607203 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:11.607174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t"] Jul 10 20:48:11.610117 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:11.610076 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d260298_f1a4_440b_84c4_c2b1adc4a6a8.slice/crio-a1ef9fc0a7a9321b229e3ae8c4ede3ef19e3e619519600a87f1b8031009c14c1 WatchSource:0}: Error finding container a1ef9fc0a7a9321b229e3ae8c4ede3ef19e3e619519600a87f1b8031009c14c1: Status 404 returned error can't find the container with id a1ef9fc0a7a9321b229e3ae8c4ede3ef19e3e619519600a87f1b8031009c14c1 Jul 10 20:48:12.323089 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:12.323052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" event={"ID":"7d260298-f1a4-440b-84c4-c2b1adc4a6a8","Type":"ContainerStarted","Data":"a1ef9fc0a7a9321b229e3ae8c4ede3ef19e3e619519600a87f1b8031009c14c1"} Jul 10 20:48:14.864416 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:14.864381 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8"] Jul 10 20:48:14.867122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:14.867104 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:14.869009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:14.868992 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hfrz9\"" Jul 10 20:48:14.880862 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:14.880841 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8"] Jul 10 20:48:14.993195 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:14.993155 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l225p\" (UniqueName: \"kubernetes.io/projected/b2e1a486-d121-40fb-a324-47c4d84fa158-kube-api-access-l225p\") pod \"authorino-operator-75cc8d5cb7-r6rf8\" (UID: \"b2e1a486-d121-40fb-a324-47c4d84fa158\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:15.094416 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:15.094369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l225p\" (UniqueName: \"kubernetes.io/projected/b2e1a486-d121-40fb-a324-47c4d84fa158-kube-api-access-l225p\") pod \"authorino-operator-75cc8d5cb7-r6rf8\" (UID: \"b2e1a486-d121-40fb-a324-47c4d84fa158\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:15.106350 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:15.106318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l225p\" (UniqueName: \"kubernetes.io/projected/b2e1a486-d121-40fb-a324-47c4d84fa158-kube-api-access-l225p\") pod \"authorino-operator-75cc8d5cb7-r6rf8\" (UID: \"b2e1a486-d121-40fb-a324-47c4d84fa158\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:15.177820 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:15.177739 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:15.330144 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:15.329868 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8"] Jul 10 20:48:15.332993 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:15.332951 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e1a486_d121_40fb_a324_47c4d84fa158.slice/crio-7a859366d8ad10a97720886f10626609793d9db354ca824c2e4eca5298e3fd6f WatchSource:0}: Error finding container 7a859366d8ad10a97720886f10626609793d9db354ca824c2e4eca5298e3fd6f: Status 404 returned error can't find the container with id 7a859366d8ad10a97720886f10626609793d9db354ca824c2e4eca5298e3fd6f Jul 10 20:48:16.297423 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:16.297389 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:16.345493 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:16.345450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" event={"ID":"b2e1a486-d121-40fb-a324-47c4d84fa158","Type":"ContainerStarted","Data":"7a859366d8ad10a97720886f10626609793d9db354ca824c2e4eca5298e3fd6f"} Jul 10 20:48:17.352696 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:17.352612 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" event={"ID":"7d260298-f1a4-440b-84c4-c2b1adc4a6a8","Type":"ContainerStarted","Data":"dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34"} Jul 10 20:48:17.353060 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:17.352698 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:17.376444 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:17.376372 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" podStartSLOduration=0.959651991 podStartE2EDuration="6.376353769s" totalImagesPullingTime="5.416701778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:17.375806417 +0000 UTC m=+627.982563031" watchObservedRunningTime="2026-07-10 20:48:17.376353769 +0000 UTC m=+627.983110352" Jul 10 20:48:19.363255 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:19.363137 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" event={"ID":"b2e1a486-d121-40fb-a324-47c4d84fa158","Type":"ContainerStarted","Data":"203e46fb85800e9448b4baaf87ab17bc5ba23f186701ae0d2d756562d2a5f6a6"} Jul 10 20:48:19.363616 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:19.363292 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:19.381155 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:19.381097 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" podStartSLOduration=1.775201718 podStartE2EDuration="5.381081608s" totalImagesPullingTime="3.60587989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:19.379204143 +0000 UTC m=+629.985960744" watchObservedRunningTime="2026-07-10 20:48:19.381081608 +0000 UTC m=+629.987838191" Jul 10 20:48:20.153409 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.153372 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl"] Jul 10 20:48:20.156730 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.156712 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.158467 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.158444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 20:48:20.158588 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.158569 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 20:48:20.158652 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.158645 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dwf2s\"" Jul 10 20:48:20.166044 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.166019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl"] Jul 10 20:48:20.243394 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.243359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.243532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.243420 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fvp\" (UniqueName: \"kubernetes.io/projected/57d40615-bb8e-45f8-8ad6-20a837d190a5-kube-api-access-t2fvp\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.243532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.243453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57d40615-bb8e-45f8-8ad6-20a837d190a5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.315942 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.315913 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-njzfb" Jul 10 20:48:20.344470 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.344441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fvp\" (UniqueName: \"kubernetes.io/projected/57d40615-bb8e-45f8-8ad6-20a837d190a5-kube-api-access-t2fvp\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.344470 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.344474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57d40615-bb8e-45f8-8ad6-20a837d190a5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.344686 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.344526 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.344686 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:20.344628 2585 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 20:48:20.344768 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:20.344710 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert podName:57d40615-bb8e-45f8-8ad6-20a837d190a5 nodeName:}" failed. No retries permitted until 2026-07-10 20:48:20.844693611 +0000 UTC m=+631.451450171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-xt7rl" (UID: "57d40615-bb8e-45f8-8ad6-20a837d190a5") : secret "plugin-serving-cert" not found Jul 10 20:48:20.345135 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.345115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/57d40615-bb8e-45f8-8ad6-20a837d190a5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.357729 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.357695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fvp\" (UniqueName: \"kubernetes.io/projected/57d40615-bb8e-45f8-8ad6-20a837d190a5-kube-api-access-t2fvp\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.848912 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.848880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:20.851526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:20.851504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/57d40615-bb8e-45f8-8ad6-20a837d190a5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xt7rl\" (UID: \"57d40615-bb8e-45f8-8ad6-20a837d190a5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:21.067123 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:21.067085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" Jul 10 20:48:21.212101 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:21.212076 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl"] Jul 10 20:48:21.213682 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:21.213641 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57d40615_bb8e_45f8_8ad6_20a837d190a5.slice/crio-3af1f5bb7fc316a42974dab73d3f4de787b5ec61a9f8e8b12488f060a7d6b059 WatchSource:0}: Error finding container 3af1f5bb7fc316a42974dab73d3f4de787b5ec61a9f8e8b12488f060a7d6b059: Status 404 returned error can't find the container with id 3af1f5bb7fc316a42974dab73d3f4de787b5ec61a9f8e8b12488f060a7d6b059 Jul 10 20:48:21.374028 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:21.373941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" event={"ID":"57d40615-bb8e-45f8-8ad6-20a837d190a5","Type":"ContainerStarted","Data":"3af1f5bb7fc316a42974dab73d3f4de787b5ec61a9f8e8b12488f060a7d6b059"} Jul 10 20:48:28.359377 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:28.359345 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:29.524423 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.524388 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt"] Jul 10 20:48:29.527856 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.527832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.541390 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.541368 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt"] Jul 10 20:48:29.631449 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.631417 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsw2b\" (UniqueName: \"kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.631626 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.631461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.732006 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.731964 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsw2b\" (UniqueName: \"kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.732177 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.732030 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.732479 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.732453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.741971 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.741939 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsw2b\" (UniqueName: \"kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:29.838941 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:29.838844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:30.255205 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.255166 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt"] Jul 10 20:48:30.257997 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.257926 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt"] Jul 10 20:48:30.269174 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.269138 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t"] Jul 10 20:48:30.269426 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.269400 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" containerName="manager" containerID="cri-o://dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34" gracePeriod=2 Jul 10 20:48:30.281163 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.281134 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t"] Jul 10 20:48:30.282641 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.282614 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:48:30.283059 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.283040 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" containerName="manager" Jul 10 20:48:30.283059 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.283059 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" containerName="manager" Jul 10 20:48:30.283320 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.283163 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" containerName="manager" Jul 10 20:48:30.286615 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.286592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.289680 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.289311 2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\": pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:30.298662 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.298593 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:48:30.300733 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.300706 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82"] Jul 10 20:48:30.300988 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.300952 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" containerName="manager" containerID="cri-o://0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc" gracePeriod=2 Jul 10 20:48:30.302886 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.302679 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:30.306772 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.306741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.312653 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.312624 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82"] Jul 10 20:48:30.314042 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.314000 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:30.317184 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.317019 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:30.319156 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.319004 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:30.319705 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.319678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:30.338501 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.338462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75bd\" (UniqueName: \"kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.338691 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.338519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwrq\" (UniqueName: \"kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.338691 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.338552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.338691 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.338639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.371565 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.371531 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-r6rf8" Jul 10 20:48:30.373279 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.373204 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:30.398707 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:30.398661 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:30.439593 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.439561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.439767 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.439634 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75bd\" (UniqueName: \"kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.439767 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.439659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwrq\" (UniqueName: \"kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.439767 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.439688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.440097 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.440072 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.440172 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.440098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.449293 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.449269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwrq\" (UniqueName: \"kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dhljz\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.449293 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.449275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75bd\" (UniqueName: \"kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jqxsl\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:30.684758 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.684663 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:30.693781 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:30.693750 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:44.082643 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.082390 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.083987 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.083948 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.102016 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.101925 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.103452 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.103423 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.104592 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.104563 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.110421 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.110400 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:44.112070 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:44.112043 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6381d24d_6629_4f6b_b26f_33c3f3141331.slice/crio-7f33a8badfed90ec1f8543f52d28e97b9124dd7971d6fe1f23b49d0c749d9a0b WatchSource:0}: Error finding container 7f33a8badfed90ec1f8543f52d28e97b9124dd7971d6fe1f23b49d0c749d9a0b: Status 404 returned error can't find the container with id 7f33a8badfed90ec1f8543f52d28e97b9124dd7971d6fe1f23b49d0c749d9a0b Jul 10 20:48:44.134963 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.134941 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:48:44.142454 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:44.142419 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e6ddb1_957a_47be_b619_e0a281e775ea.slice/crio-4b9db0554609cfb3d01170cbe1c185cf3105d02ee6a0fcd7377844510e1bb841 WatchSource:0}: Error finding container 4b9db0554609cfb3d01170cbe1c185cf3105d02ee6a0fcd7377844510e1bb841: Status 404 returned error can't find the container with id 4b9db0554609cfb3d01170cbe1c185cf3105d02ee6a0fcd7377844510e1bb841 Jul 10 20:48:44.171172 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.171144 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j27rl\" (UniqueName: \"kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl\") pod \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " Jul 10 20:48:44.171311 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.171212 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume\") pod \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\" (UID: \"7d260298-f1a4-440b-84c4-c2b1adc4a6a8\") " Jul 10 20:48:44.171378 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.171330 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h858h\" (UniqueName: \"kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h\") pod \"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea\" (UID: \"fd291ebb-fdc3-452c-adcc-aa9b9afd4bea\") " Jul 10 20:48:44.172550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.171995 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7d260298-f1a4-440b-84c4-c2b1adc4a6a8" (UID: "7d260298-f1a4-440b-84c4-c2b1adc4a6a8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:48:44.173614 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.173573 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl" (OuterVolumeSpecName: "kube-api-access-j27rl") pod "7d260298-f1a4-440b-84c4-c2b1adc4a6a8" (UID: "7d260298-f1a4-440b-84c4-c2b1adc4a6a8"). InnerVolumeSpecName "kube-api-access-j27rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:48:44.173768 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.173738 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h" (OuterVolumeSpecName: "kube-api-access-h858h") pod "fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" (UID: "fd291ebb-fdc3-452c-adcc-aa9b9afd4bea"). InnerVolumeSpecName "kube-api-access-h858h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:48:44.272583 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.272542 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:44.272583 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.272570 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h858h\" (UniqueName: \"kubernetes.io/projected/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea-kube-api-access-h858h\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:44.272583 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.272579 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27rl\" (UniqueName: \"kubernetes.io/projected/7d260298-f1a4-440b-84c4-c2b1adc4a6a8-kube-api-access-j27rl\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:44.493876 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.493832 2585 generic.go:356] "Generic (PLEG): container finished" podID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" containerID="0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc" exitCode=0 Jul 10 20:48:44.494050 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.493900 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.494050 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.493947 2585 scope.go:117] "RemoveContainer" containerID="0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc" Jul 10 20:48:44.495888 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.495852 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.496009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.495985 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" event={"ID":"e6e6ddb1-957a-47be-b619-e0a281e775ea","Type":"ContainerStarted","Data":"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d"} Jul 10 20:48:44.496070 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.496038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" event={"ID":"e6e6ddb1-957a-47be-b619-e0a281e775ea","Type":"ContainerStarted","Data":"4b9db0554609cfb3d01170cbe1c185cf3105d02ee6a0fcd7377844510e1bb841"} Jul 10 20:48:44.496175 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.496160 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:44.497584 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.497157 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.497763 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.497738 2585 generic.go:356] "Generic (PLEG): container finished" podID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" containerID="dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34" exitCode=0 Jul 10 20:48:44.497881 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.497868 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.499467 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.499338 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.502649 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.502080 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" event={"ID":"57d40615-bb8e-45f8-8ad6-20a837d190a5","Type":"ContainerStarted","Data":"85136a7258d78d95c58809506543a0adf2842c077e78a2ad7c58bbe6dfcaff7a"} Jul 10 20:48:44.504657 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.504632 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" event={"ID":"6381d24d-6629-4f6b-b26f-33c3f3141331","Type":"ContainerStarted","Data":"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873"} Jul 10 20:48:44.504755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.504688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" event={"ID":"6381d24d-6629-4f6b-b26f-33c3f3141331","Type":"ContainerStarted","Data":"7f33a8badfed90ec1f8543f52d28e97b9124dd7971d6fe1f23b49d0c749d9a0b"} Jul 10 20:48:44.504755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.504720 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:44.507334 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.507280 2585 scope.go:117] "RemoveContainer" containerID="0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc" Jul 10 20:48:44.507541 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.507518 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc\": container with ID starting with 0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc not found: ID does not exist" containerID="0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc" Jul 10 20:48:44.507622 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.507550 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc"} err="failed to get container status \"0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc\": rpc error: code = NotFound desc = could not find container \"0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc\": container with ID starting with 0b633920b659d645d4c22d9a1791f6b8b22bd102dedca0b93f3ba74b6cf8c6cc not found: ID does not exist" Jul 10 20:48:44.507622 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.507566 2585 scope.go:117] "RemoveContainer" containerID="dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34" Jul 10 20:48:44.516042 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.516027 2585 scope.go:117] "RemoveContainer" containerID="dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34" Jul 10 20:48:44.516293 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.516273 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34\": container with ID starting with dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34 not found: ID does not exist" containerID="dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34" Jul 10 20:48:44.516368 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.516298 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34"} err="failed to get container status \"dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34\": rpc error: code = NotFound desc = could not find container \"dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34\": container with ID starting with dfefc8aaa4f2801f39ca0e4196d237f2a2c3baaee3ddc7ea7516fc752978bf34 not found: ID does not exist" Jul 10 20:48:44.518285 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.518218 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" podStartSLOduration=14.518205677 podStartE2EDuration="14.518205677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:44.517705814 +0000 UTC m=+655.124462421" watchObservedRunningTime="2026-07-10 20:48:44.518205677 +0000 UTC m=+655.124962259" Jul 10 20:48:44.519114 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.519090 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.520525 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.520505 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fxs82\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fxs82" Jul 10 20:48:44.521830 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:44.521810 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-cmz6t\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-cmz6t" Jul 10 20:48:44.549121 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.549064 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" podStartSLOduration=14.549046089 podStartE2EDuration="14.549046089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:44.547409654 +0000 UTC m=+655.154166255" watchObservedRunningTime="2026-07-10 20:48:44.549046089 +0000 UTC m=+655.155802671" Jul 10 20:48:44.566616 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:44.566564 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xt7rl" podStartSLOduration=1.7917630610000002 podStartE2EDuration="24.566547061s" totalImagesPullingTime="22.774784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:44.564961868 +0000 UTC m=+655.171718450" watchObservedRunningTime="2026-07-10 20:48:44.566547061 +0000 UTC m=+655.173303659" Jul 10 20:48:45.984805 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:45.984775 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d260298-f1a4-440b-84c4-c2b1adc4a6a8" path="/var/lib/kubelet/pods/7d260298-f1a4-440b-84c4-c2b1adc4a6a8/volumes" Jul 10 20:48:45.985201 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:45.985079 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" path="/var/lib/kubelet/pods/fd291ebb-fdc3-452c-adcc-aa9b9afd4bea/volumes" Jul 10 20:48:46.482145 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:46.482110 2585 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-bb9wt_kuadrant-system_140dc9e6-8040-426e-b537-5d5f30a95313_0(e551f1d445670104047069b279432248ba76ebe348cedf55d3717a4f5988216c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-bb9wt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 10 20:48:46.482225 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:46.482172 2585 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-bb9wt_kuadrant-system_140dc9e6-8040-426e-b537-5d5f30a95313_0(e551f1d445670104047069b279432248ba76ebe348cedf55d3717a4f5988216c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-bb9wt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:46.515721 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.515691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:46.517444 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:46.517404 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="140dc9e6-8040-426e-b537-5d5f30a95313" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:46.522493 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.522469 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:46.524016 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:46.523993 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="140dc9e6-8040-426e-b537-5d5f30a95313" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:46.594439 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.594405 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume\") pod \"140dc9e6-8040-426e-b537-5d5f30a95313\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " Jul 10 20:48:46.594606 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.594490 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsw2b\" (UniqueName: \"kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b\") pod \"140dc9e6-8040-426e-b537-5d5f30a95313\" (UID: \"140dc9e6-8040-426e-b537-5d5f30a95313\") " Jul 10 20:48:46.594699 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.594677 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "140dc9e6-8040-426e-b537-5d5f30a95313" (UID: "140dc9e6-8040-426e-b537-5d5f30a95313"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:48:46.596769 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.596741 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b" (OuterVolumeSpecName: "kube-api-access-bsw2b") pod "140dc9e6-8040-426e-b537-5d5f30a95313" (UID: "140dc9e6-8040-426e-b537-5d5f30a95313"). InnerVolumeSpecName "kube-api-access-bsw2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:48:46.695682 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.695642 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/140dc9e6-8040-426e-b537-5d5f30a95313-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:46.695682 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:46.695673 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsw2b\" (UniqueName: \"kubernetes.io/projected/140dc9e6-8040-426e-b537-5d5f30a95313-kube-api-access-bsw2b\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:47.520014 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:47.519978 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:47.522087 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:47.522059 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="140dc9e6-8040-426e-b537-5d5f30a95313" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:47.527116 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:47.527087 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bb9wt\" is forbidden: User \"system:node:ip-10-0-142-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-194.ec2.internal' and this object" podUID="140dc9e6-8040-426e-b537-5d5f30a95313" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bb9wt" Jul 10 20:48:47.985049 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:47.985006 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140dc9e6-8040-426e-b537-5d5f30a95313" path="/var/lib/kubelet/pods/140dc9e6-8040-426e-b537-5d5f30a95313/volumes" Jul 10 20:48:55.513445 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.513415 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:48:55.513986 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.513486 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:55.576750 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.576715 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:55.576931 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.576893 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" podUID="6381d24d-6629-4f6b-b26f-33c3f3141331" containerName="manager" containerID="cri-o://9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873" gracePeriod=10 Jul 10 20:48:55.829303 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.829280 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:55.860064 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860034 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 20:48:55.860491 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860476 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" containerName="manager" Jul 10 20:48:55.860565 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860494 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" containerName="manager" Jul 10 20:48:55.860565 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860504 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6381d24d-6629-4f6b-b26f-33c3f3141331" containerName="manager" Jul 10 20:48:55.860565 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860509 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6381d24d-6629-4f6b-b26f-33c3f3141331" containerName="manager" Jul 10 20:48:55.860659 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860583 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd291ebb-fdc3-452c-adcc-aa9b9afd4bea" containerName="manager" Jul 10 20:48:55.860659 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.860591 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6381d24d-6629-4f6b-b26f-33c3f3141331" containerName="manager" Jul 10 20:48:55.862531 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.862515 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:55.875088 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.875062 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 20:48:55.977966 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.977931 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume\") pod \"6381d24d-6629-4f6b-b26f-33c3f3141331\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " Jul 10 20:48:55.977966 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.977968 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t75bd\" (UniqueName: \"kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd\") pod \"6381d24d-6629-4f6b-b26f-33c3f3141331\" (UID: \"6381d24d-6629-4f6b-b26f-33c3f3141331\") " Jul 10 20:48:55.978169 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.978113 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24mq\" (UniqueName: \"kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:55.978205 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.978183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:55.978447 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.978420 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6381d24d-6629-4f6b-b26f-33c3f3141331" (UID: "6381d24d-6629-4f6b-b26f-33c3f3141331"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:48:55.980299 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:55.980271 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd" (OuterVolumeSpecName: "kube-api-access-t75bd") pod "6381d24d-6629-4f6b-b26f-33c3f3141331" (UID: "6381d24d-6629-4f6b-b26f-33c3f3141331"). InnerVolumeSpecName "kube-api-access-t75bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:48:56.079745 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.079629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24mq\" (UniqueName: \"kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.079745 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.079721 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.079993 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.079768 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6381d24d-6629-4f6b-b26f-33c3f3141331-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:56.079993 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.079777 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75bd\" (UniqueName: \"kubernetes.io/projected/6381d24d-6629-4f6b-b26f-33c3f3141331-kube-api-access-t75bd\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:48:56.080139 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.080118 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.087502 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.087471 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24mq\" (UniqueName: \"kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq\") pod \"kuadrant-operator-controller-manager-86878c769f-5rttf\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.174291 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.174191 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.319675 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.319529 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 20:48:56.322410 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:48:56.322383 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081c4038_7afb_41d1_b378_1098ee99d918.slice/crio-e1d0d67e442fc66ba2aa0726a5aed02b712b9dae6a781ce67db5b03087630c80 WatchSource:0}: Error finding container e1d0d67e442fc66ba2aa0726a5aed02b712b9dae6a781ce67db5b03087630c80: Status 404 returned error can't find the container with id e1d0d67e442fc66ba2aa0726a5aed02b712b9dae6a781ce67db5b03087630c80 Jul 10 20:48:56.561815 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.561780 2585 generic.go:356] "Generic (PLEG): container finished" podID="6381d24d-6629-4f6b-b26f-33c3f3141331" containerID="9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873" exitCode=0 Jul 10 20:48:56.562314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.561825 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" Jul 10 20:48:56.562314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.561864 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" event={"ID":"6381d24d-6629-4f6b-b26f-33c3f3141331","Type":"ContainerDied","Data":"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873"} Jul 10 20:48:56.562314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.561927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl" event={"ID":"6381d24d-6629-4f6b-b26f-33c3f3141331","Type":"ContainerDied","Data":"7f33a8badfed90ec1f8543f52d28e97b9124dd7971d6fe1f23b49d0c749d9a0b"} Jul 10 20:48:56.562314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.561961 2585 scope.go:117] "RemoveContainer" containerID="9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873" Jul 10 20:48:56.563926 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.563876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" event={"ID":"081c4038-7afb-41d1-b378-1098ee99d918","Type":"ContainerStarted","Data":"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8"} Jul 10 20:48:56.564104 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.563927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" event={"ID":"081c4038-7afb-41d1-b378-1098ee99d918","Type":"ContainerStarted","Data":"e1d0d67e442fc66ba2aa0726a5aed02b712b9dae6a781ce67db5b03087630c80"} Jul 10 20:48:56.564104 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.563968 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:48:56.570772 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.570753 2585 scope.go:117] "RemoveContainer" containerID="9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873" Jul 10 20:48:56.571031 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:48:56.571012 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873\": container with ID starting with 9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873 not found: ID does not exist" containerID="9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873" Jul 10 20:48:56.571083 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.571041 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873"} err="failed to get container status \"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873\": rpc error: code = NotFound desc = could not find container \"9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873\": container with ID starting with 9c6bd24ac243dbc91366226e2cff40b9cd4cafa20d74cd89d2a2db7b78bd4873 not found: ID does not exist" Jul 10 20:48:56.579080 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.579058 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:56.582865 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.582841 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jqxsl"] Jul 10 20:48:56.599261 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:56.599136 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" podStartSLOduration=1.599119463 podStartE2EDuration="1.599119463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:56.597758061 +0000 UTC m=+667.204514680" watchObservedRunningTime="2026-07-10 20:48:56.599119463 +0000 UTC m=+667.205876044" Jul 10 20:48:57.985083 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:48:57.985051 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6381d24d-6629-4f6b-b26f-33c3f3141331" path="/var/lib/kubelet/pods/6381d24d-6629-4f6b-b26f-33c3f3141331/volumes" Jul 10 20:49:07.571318 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:07.571283 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 20:49:07.619033 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:07.618998 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:49:07.619219 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:07.619195 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" podUID="e6e6ddb1-957a-47be-b619-e0a281e775ea" containerName="manager" containerID="cri-o://1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d" gracePeriod=10 Jul 10 20:49:08.398356 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.398326 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:49:08.490430 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.490401 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbwrq\" (UniqueName: \"kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq\") pod \"e6e6ddb1-957a-47be-b619-e0a281e775ea\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " Jul 10 20:49:08.490578 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.490448 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume\") pod \"e6e6ddb1-957a-47be-b619-e0a281e775ea\" (UID: \"e6e6ddb1-957a-47be-b619-e0a281e775ea\") " Jul 10 20:49:08.490840 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.490815 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e6e6ddb1-957a-47be-b619-e0a281e775ea" (UID: "e6e6ddb1-957a-47be-b619-e0a281e775ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:49:08.492638 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.492615 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq" (OuterVolumeSpecName: "kube-api-access-jbwrq") pod "e6e6ddb1-957a-47be-b619-e0a281e775ea" (UID: "e6e6ddb1-957a-47be-b619-e0a281e775ea"). InnerVolumeSpecName "kube-api-access-jbwrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:49:08.591357 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.591281 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbwrq\" (UniqueName: \"kubernetes.io/projected/e6e6ddb1-957a-47be-b619-e0a281e775ea-kube-api-access-jbwrq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:49:08.591357 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.591311 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6e6ddb1-957a-47be-b619-e0a281e775ea-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:49:08.622695 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.622589 2585 generic.go:356] "Generic (PLEG): container finished" podID="e6e6ddb1-957a-47be-b619-e0a281e775ea" containerID="1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d" exitCode=0 Jul 10 20:49:08.622695 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.622640 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" Jul 10 20:49:08.622695 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.622685 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" event={"ID":"e6e6ddb1-957a-47be-b619-e0a281e775ea","Type":"ContainerDied","Data":"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d"} Jul 10 20:49:08.622928 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.622755 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz" event={"ID":"e6e6ddb1-957a-47be-b619-e0a281e775ea","Type":"ContainerDied","Data":"4b9db0554609cfb3d01170cbe1c185cf3105d02ee6a0fcd7377844510e1bb841"} Jul 10 20:49:08.622928 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.622788 2585 scope.go:117] "RemoveContainer" containerID="1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d" Jul 10 20:49:08.632028 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.632011 2585 scope.go:117] "RemoveContainer" containerID="1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d" Jul 10 20:49:08.632302 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:49:08.632281 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d\": container with ID starting with 1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d not found: ID does not exist" containerID="1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d" Jul 10 20:49:08.632402 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.632310 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d"} err="failed to get container status \"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d\": rpc error: code = NotFound desc = could not find container \"1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d\": container with ID starting with 1305df4700abe77177ed1a9a1d8452b407978d2337822224e8ae2d1e87bebf1d not found: ID does not exist" Jul 10 20:49:08.642853 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.642829 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:49:08.648972 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:08.648952 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dhljz"] Jul 10 20:49:09.985272 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:09.985221 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e6ddb1-957a-47be-b619-e0a281e775ea" path="/var/lib/kubelet/pods/e6e6ddb1-957a-47be-b619-e0a281e775ea/volumes" Jul 10 20:49:24.248415 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.248379 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:24.248862 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.248693 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e6ddb1-957a-47be-b619-e0a281e775ea" containerName="manager" Jul 10 20:49:24.248862 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.248705 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e6ddb1-957a-47be-b619-e0a281e775ea" containerName="manager" Jul 10 20:49:24.248862 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.248772 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e6ddb1-957a-47be-b619-e0a281e775ea" containerName="manager" Jul 10 20:49:24.292660 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.292622 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:24.292820 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.292722 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.294665 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.294644 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 20:49:24.346045 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.346012 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:24.423734 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.423704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.423920 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.423781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ljc\" (UniqueName: \"kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.525248 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.525154 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ljc\" (UniqueName: \"kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.525248 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.525205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.525802 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.525783 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.535014 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.534988 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ljc\" (UniqueName: \"kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc\") pod \"limitador-limitador-64c94fd64f-bv2zz\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.602797 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.602758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:24.739741 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.739717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:24.742073 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:49:24.742035 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8b2a1a_6bd1_472f_80bb_b9ebc69c4e46.slice/crio-1f85b3f7dc7d18764abb0320a5f19e67d6c18e4f2c0e002c74fdf3cebef11f5a WatchSource:0}: Error finding container 1f85b3f7dc7d18764abb0320a5f19e67d6c18e4f2c0e002c74fdf3cebef11f5a: Status 404 returned error can't find the container with id 1f85b3f7dc7d18764abb0320a5f19e67d6c18e4f2c0e002c74fdf3cebef11f5a Jul 10 20:49:24.744707 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:24.744691 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:49:25.049476 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.049423 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:25.053819 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.053802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:25.055945 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.055924 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-szm5d\"" Jul 10 20:49:25.060068 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.060045 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:25.130598 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.130565 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5zg\" (UniqueName: \"kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg\") pod \"authorino-58cd955675-bvqp9\" (UID: \"aea309d4-8bea-47df-8b05-5f035df0ca2c\") " pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:25.231613 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.231576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5zg\" (UniqueName: \"kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg\") pod \"authorino-58cd955675-bvqp9\" (UID: \"aea309d4-8bea-47df-8b05-5f035df0ca2c\") " pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:25.244594 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.244564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5zg\" (UniqueName: \"kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg\") pod \"authorino-58cd955675-bvqp9\" (UID: \"aea309d4-8bea-47df-8b05-5f035df0ca2c\") " pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:25.365675 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.365586 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:25.538137 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.538081 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:25.540660 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:49:25.540628 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaea309d4_8bea_47df_8b05_5f035df0ca2c.slice/crio-cb66255a2733b7e2582f0b646b058c0919fe60252d757e3c13815b16174ba98b WatchSource:0}: Error finding container cb66255a2733b7e2582f0b646b058c0919fe60252d757e3c13815b16174ba98b: Status 404 returned error can't find the container with id cb66255a2733b7e2582f0b646b058c0919fe60252d757e3c13815b16174ba98b Jul 10 20:49:25.716728 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.716691 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" event={"ID":"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46","Type":"ContainerStarted","Data":"1f85b3f7dc7d18764abb0320a5f19e67d6c18e4f2c0e002c74fdf3cebef11f5a"} Jul 10 20:49:25.718323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:25.718289 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bvqp9" event={"ID":"aea309d4-8bea-47df-8b05-5f035df0ca2c","Type":"ContainerStarted","Data":"cb66255a2733b7e2582f0b646b058c0919fe60252d757e3c13815b16174ba98b"} Jul 10 20:49:29.037314 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.037270 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:29.742453 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.742412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" event={"ID":"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46","Type":"ContainerStarted","Data":"e22e14edaa1f2410b0325114713745f5bf4c11b9e1d1f2ac62e10ce7d23c6119"} Jul 10 20:49:29.742689 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.742522 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:29.743981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.743959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bvqp9" event={"ID":"aea309d4-8bea-47df-8b05-5f035df0ca2c","Type":"ContainerStarted","Data":"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264"} Jul 10 20:49:29.743981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.743968 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-bvqp9" podUID="aea309d4-8bea-47df-8b05-5f035df0ca2c" containerName="authorino" containerID="cri-o://e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264" gracePeriod=30 Jul 10 20:49:29.760197 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.760152 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" podStartSLOduration=1.176658432 podStartE2EDuration="5.760136964s" totalImagesPullingTime="4.583478532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:29.758037676 +0000 UTC m=+700.364794258" watchObservedRunningTime="2026-07-10 20:49:29.760136964 +0000 UTC m=+700.366893581" Jul 10 20:49:29.774657 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:29.774613 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-bvqp9" podStartSLOduration=0.981913781 podStartE2EDuration="4.774596453s" totalImagesPullingTime="3.792682672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:29.773843896 +0000 UTC m=+700.380600475" watchObservedRunningTime="2026-07-10 20:49:29.774596453 +0000 UTC m=+700.381353034" Jul 10 20:49:30.009124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.009096 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:30.078597 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.078561 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m5zg\" (UniqueName: \"kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg\") pod \"aea309d4-8bea-47df-8b05-5f035df0ca2c\" (UID: \"aea309d4-8bea-47df-8b05-5f035df0ca2c\") " Jul 10 20:49:30.080750 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.080727 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg" (OuterVolumeSpecName: "kube-api-access-5m5zg") pod "aea309d4-8bea-47df-8b05-5f035df0ca2c" (UID: "aea309d4-8bea-47df-8b05-5f035df0ca2c"). InnerVolumeSpecName "kube-api-access-5m5zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:49:30.179658 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.179616 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5zg\" (UniqueName: \"kubernetes.io/projected/aea309d4-8bea-47df-8b05-5f035df0ca2c-kube-api-access-5m5zg\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:49:30.749545 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.749513 2585 generic.go:356] "Generic (PLEG): container finished" podID="aea309d4-8bea-47df-8b05-5f035df0ca2c" containerID="e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264" exitCode=0 Jul 10 20:49:30.749732 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.749556 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-bvqp9" Jul 10 20:49:30.749732 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.749601 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bvqp9" event={"ID":"aea309d4-8bea-47df-8b05-5f035df0ca2c","Type":"ContainerDied","Data":"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264"} Jul 10 20:49:30.749732 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.749639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-bvqp9" event={"ID":"aea309d4-8bea-47df-8b05-5f035df0ca2c","Type":"ContainerDied","Data":"cb66255a2733b7e2582f0b646b058c0919fe60252d757e3c13815b16174ba98b"} Jul 10 20:49:30.749732 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.749655 2585 scope.go:117] "RemoveContainer" containerID="e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264" Jul 10 20:49:30.762312 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.762291 2585 scope.go:117] "RemoveContainer" containerID="e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264" Jul 10 20:49:30.762617 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:49:30.762597 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264\": container with ID starting with e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264 not found: ID does not exist" containerID="e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264" Jul 10 20:49:30.762684 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.762625 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264"} err="failed to get container status \"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264\": rpc error: code = NotFound desc = could not find container \"e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264\": container with ID starting with e856fe2580af1063948cd1896d540a388e0dfd9d58c6496fb6bb10a272ec8264 not found: ID does not exist" Jul 10 20:49:30.772313 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.772290 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:30.776311 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:30.776289 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-bvqp9"] Jul 10 20:49:31.984716 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:31.984678 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea309d4-8bea-47df-8b05-5f035df0ca2c" path="/var/lib/kubelet/pods/aea309d4-8bea-47df-8b05-5f035df0ca2c/volumes" Jul 10 20:49:38.924224 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:38.924192 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:38.924620 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:38.924408 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" podUID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" containerName="limitador" containerID="cri-o://e22e14edaa1f2410b0325114713745f5bf4c11b9e1d1f2ac62e10ce7d23c6119" gracePeriod=30 Jul 10 20:49:38.925094 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:38.925029 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:39.793571 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.793348 2585 generic.go:356] "Generic (PLEG): container finished" podID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" containerID="e22e14edaa1f2410b0325114713745f5bf4c11b9e1d1f2ac62e10ce7d23c6119" exitCode=0 Jul 10 20:49:39.793571 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.793467 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" event={"ID":"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46","Type":"ContainerDied","Data":"e22e14edaa1f2410b0325114713745f5bf4c11b9e1d1f2ac62e10ce7d23c6119"} Jul 10 20:49:39.876165 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.876136 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:39.965400 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.965363 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file\") pod \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " Jul 10 20:49:39.965839 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.965414 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66ljc\" (UniqueName: \"kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc\") pod \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\" (UID: \"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46\") " Jul 10 20:49:39.965839 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.965715 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file" (OuterVolumeSpecName: "config-file") pod "8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" (UID: "8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:49:39.967739 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:39.967706 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc" (OuterVolumeSpecName: "kube-api-access-66ljc") pod "8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" (UID: "8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46"). InnerVolumeSpecName "kube-api-access-66ljc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:49:40.066952 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.066901 2585 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-config-file\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:49:40.066952 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.066944 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ljc\" (UniqueName: \"kubernetes.io/projected/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46-kube-api-access-66ljc\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:49:40.773394 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773356 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh"] Jul 10 20:49:40.773825 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773810 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea309d4-8bea-47df-8b05-5f035df0ca2c" containerName="authorino" Jul 10 20:49:40.773875 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773827 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea309d4-8bea-47df-8b05-5f035df0ca2c" containerName="authorino" Jul 10 20:49:40.773875 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773837 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" containerName="limitador" Jul 10 20:49:40.773875 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773842 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" containerName="limitador" Jul 10 20:49:40.773976 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773896 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" containerName="limitador" Jul 10 20:49:40.773976 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.773903 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea309d4-8bea-47df-8b05-5f035df0ca2c" containerName="authorino" Jul 10 20:49:40.776932 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.776912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.779066 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.779042 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 20:49:40.779181 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.779047 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6l84r\"" Jul 10 20:49:40.779181 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.779048 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 20:49:40.779181 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.779054 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 20:49:40.788968 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.788945 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh"] Jul 10 20:49:40.800166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.800139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" event={"ID":"8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46","Type":"ContainerDied","Data":"1f85b3f7dc7d18764abb0320a5f19e67d6c18e4f2c0e002c74fdf3cebef11f5a"} Jul 10 20:49:40.800317 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.800190 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bv2zz" Jul 10 20:49:40.800317 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.800198 2585 scope.go:117] "RemoveContainer" containerID="e22e14edaa1f2410b0325114713745f5bf4c11b9e1d1f2ac62e10ce7d23c6119" Jul 10 20:49:40.814609 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.814585 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:40.817889 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.817869 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bv2zz"] Jul 10 20:49:40.874503 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.874469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn894\" (UniqueName: \"kubernetes.io/projected/3e00795f-8e91-48a7-bce8-389c05d63636-kube-api-access-vn894\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.874503 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.874506 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e00795f-8e91-48a7-bce8-389c05d63636-data\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.975394 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.975355 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e00795f-8e91-48a7-bce8-389c05d63636-data\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.975770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.975479 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn894\" (UniqueName: \"kubernetes.io/projected/3e00795f-8e91-48a7-bce8-389c05d63636-kube-api-access-vn894\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.975770 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.975740 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e00795f-8e91-48a7-bce8-389c05d63636-data\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:40.983875 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:40.983848 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn894\" (UniqueName: \"kubernetes.io/projected/3e00795f-8e91-48a7-bce8-389c05d63636-kube-api-access-vn894\") pod \"postgres-69fb4594cf-hwtmh\" (UID: \"3e00795f-8e91-48a7-bce8-389c05d63636\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:41.087574 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:41.087484 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:41.226562 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:41.226525 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh"] Jul 10 20:49:41.232886 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:49:41.232850 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e00795f_8e91_48a7_bce8_389c05d63636.slice/crio-c0063e44ee123345b9283c46d0d8d93a7dd7b37cdfe1ed80c3e1785d115e053c WatchSource:0}: Error finding container c0063e44ee123345b9283c46d0d8d93a7dd7b37cdfe1ed80c3e1785d115e053c: Status 404 returned error can't find the container with id c0063e44ee123345b9283c46d0d8d93a7dd7b37cdfe1ed80c3e1785d115e053c Jul 10 20:49:41.806846 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:41.806809 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" event={"ID":"3e00795f-8e91-48a7-bce8-389c05d63636","Type":"ContainerStarted","Data":"c0063e44ee123345b9283c46d0d8d93a7dd7b37cdfe1ed80c3e1785d115e053c"} Jul 10 20:49:41.984800 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:41.984766 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46" path="/var/lib/kubelet/pods/8b8b2a1a-6bd1-472f-80bb-b9ebc69c4e46/volumes" Jul 10 20:49:46.835486 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:46.835438 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" event={"ID":"3e00795f-8e91-48a7-bce8-389c05d63636","Type":"ContainerStarted","Data":"151b1c75f5c8f5ddfe6171f9c12767f7a1feb88353ea99d916daac1aa0af2776"} Jul 10 20:49:46.835920 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:46.835520 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:49:46.852582 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:46.852531 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" podStartSLOduration=1.826577672 podStartE2EDuration="6.852513865s" totalImagesPullingTime="5.025936193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:46.85073208 +0000 UTC m=+717.457488679" watchObservedRunningTime="2026-07-10 20:49:46.852513865 +0000 UTC m=+717.459270447" Jul 10 20:49:52.870486 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:49:52.870456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwtmh" Jul 10 20:50:00.029620 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.029585 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:00.172973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.172937 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:00.173137 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.173062 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.175276 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.175249 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 20:50:00.175383 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.175325 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wsbgn\"" Jul 10 20:50:00.175436 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.175420 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 20:50:00.200610 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.200577 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:50:00.221589 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.221555 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:50:00.221589 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.221588 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.231286 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.231260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.231429 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.231301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctct\" (UniqueName: \"kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.332576 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.332490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.332576 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.332531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.332576 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.332551 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tfm\" (UniqueName: \"kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.332824 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.332586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctct\" (UniqueName: \"kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.332824 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:50:00.332606 2585 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 20:50:00.332824 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:50:00.332666 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert podName:7c6ed47b-dbe3-40fa-b134-43227cbb3872 nodeName:}" failed. No retries permitted until 2026-07-10 20:50:00.832643649 +0000 UTC m=+731.439400217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert") pod "maas-controller-6d9b4b8b8c-k8687" (UID: "7c6ed47b-dbe3-40fa-b134-43227cbb3872") : secret "maas-controller-webhook-cert" not found Jul 10 20:50:00.341361 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.341328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctct\" (UniqueName: \"kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.433647 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.433612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.433647 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.433649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tfm\" (UniqueName: \"kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.436122 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.436092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.450268 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.450217 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tfm\" (UniqueName: \"kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm\") pod \"maas-controller-74ff47775-xnvll\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.484604 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.484575 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:00.648990 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.648963 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:00.650829 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:00.650800 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf342604a_3cb4_4691_a8d5_e78344e0235e.slice/crio-b8e94dce4d3ad462bd8dc143671bfe2e1681fdd70abce38a793290333235b437 WatchSource:0}: Error finding container b8e94dce4d3ad462bd8dc143671bfe2e1681fdd70abce38a793290333235b437: Status 404 returned error can't find the container with id b8e94dce4d3ad462bd8dc143671bfe2e1681fdd70abce38a793290333235b437 Jul 10 20:50:00.836561 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.836523 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.839022 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.839003 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-k8687\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:00.897501 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:00.897428 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ff47775-xnvll" event={"ID":"f342604a-3cb4-4691-a8d5-e78344e0235e","Type":"ContainerStarted","Data":"b8e94dce4d3ad462bd8dc143671bfe2e1681fdd70abce38a793290333235b437"} Jul 10 20:50:01.132549 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:01.132511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:01.299929 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:01.299906 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:50:01.301435 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:01.301410 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6ed47b_dbe3_40fa_b134_43227cbb3872.slice/crio-f80d8b96cabc230a0b1d51ad95282dd2574c525f601a063da64189164a2df389 WatchSource:0}: Error finding container f80d8b96cabc230a0b1d51ad95282dd2574c525f601a063da64189164a2df389: Status 404 returned error can't find the container with id f80d8b96cabc230a0b1d51ad95282dd2574c525f601a063da64189164a2df389 Jul 10 20:50:01.905771 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:01.905722 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" event={"ID":"7c6ed47b-dbe3-40fa-b134-43227cbb3872","Type":"ContainerStarted","Data":"f80d8b96cabc230a0b1d51ad95282dd2574c525f601a063da64189164a2df389"} Jul 10 20:50:03.916302 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.916166 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" event={"ID":"7c6ed47b-dbe3-40fa-b134-43227cbb3872","Type":"ContainerStarted","Data":"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b"} Jul 10 20:50:03.916302 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.916277 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:03.917622 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.917602 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ff47775-xnvll" event={"ID":"f342604a-3cb4-4691-a8d5-e78344e0235e","Type":"ContainerStarted","Data":"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22"} Jul 10 20:50:03.917700 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.917647 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:03.932199 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.932156 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" podStartSLOduration=1.7827816589999999 podStartE2EDuration="3.932141966s" totalImagesPullingTime="2.149360307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:03.931082132 +0000 UTC m=+734.537838715" watchObservedRunningTime="2026-07-10 20:50:03.932141966 +0000 UTC m=+734.538898525" Jul 10 20:50:03.946917 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:03.946861 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74ff47775-xnvll" podStartSLOduration=1.136468209 podStartE2EDuration="3.946847598s" totalImagesPullingTime="2.810379389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:03.945359333 +0000 UTC m=+734.552115934" watchObservedRunningTime="2026-07-10 20:50:03.946847598 +0000 UTC m=+734.553604205" Jul 10 20:50:14.924659 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:14.924627 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:14.925037 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:14.924699 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:50:14.991738 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:14.986647 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:14.991738 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:14.986912 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74ff47775-xnvll" podUID="f342604a-3cb4-4691-a8d5-e78344e0235e" containerName="manager" containerID="cri-o://aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22" gracePeriod=30 Jul 10 20:50:15.246894 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.246860 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:15.364890 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.364852 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert\") pod \"f342604a-3cb4-4691-a8d5-e78344e0235e\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " Jul 10 20:50:15.365056 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.364904 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2tfm\" (UniqueName: \"kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm\") pod \"f342604a-3cb4-4691-a8d5-e78344e0235e\" (UID: \"f342604a-3cb4-4691-a8d5-e78344e0235e\") " Jul 10 20:50:15.367280 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.367222 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm" (OuterVolumeSpecName: "kube-api-access-v2tfm") pod "f342604a-3cb4-4691-a8d5-e78344e0235e" (UID: "f342604a-3cb4-4691-a8d5-e78344e0235e"). InnerVolumeSpecName "kube-api-access-v2tfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:50:15.367392 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.367286 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f342604a-3cb4-4691-a8d5-e78344e0235e" (UID: "f342604a-3cb4-4691-a8d5-e78344e0235e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:50:15.465654 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.465616 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f342604a-3cb4-4691-a8d5-e78344e0235e-webhook-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:50:15.465654 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.465647 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tfm\" (UniqueName: \"kubernetes.io/projected/f342604a-3cb4-4691-a8d5-e78344e0235e-kube-api-access-v2tfm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:50:15.976298 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.976262 2585 generic.go:356] "Generic (PLEG): container finished" podID="f342604a-3cb4-4691-a8d5-e78344e0235e" containerID="aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22" exitCode=0 Jul 10 20:50:15.976752 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.976310 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74ff47775-xnvll" Jul 10 20:50:15.976752 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.976334 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ff47775-xnvll" event={"ID":"f342604a-3cb4-4691-a8d5-e78344e0235e","Type":"ContainerDied","Data":"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22"} Jul 10 20:50:15.976752 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.976379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74ff47775-xnvll" event={"ID":"f342604a-3cb4-4691-a8d5-e78344e0235e","Type":"ContainerDied","Data":"b8e94dce4d3ad462bd8dc143671bfe2e1681fdd70abce38a793290333235b437"} Jul 10 20:50:15.976752 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.976406 2585 scope.go:117] "RemoveContainer" containerID="aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22" Jul 10 20:50:15.986922 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.986896 2585 scope.go:117] "RemoveContainer" containerID="aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22" Jul 10 20:50:15.987219 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:50:15.987197 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22\": container with ID starting with aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22 not found: ID does not exist" containerID="aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22" Jul 10 20:50:15.987307 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.987255 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22"} err="failed to get container status \"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22\": rpc error: code = NotFound desc = could not find container \"aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22\": container with ID starting with aef287518db3ffcce174b623f4b1a79dcb920df235c012e81d9353186fbc2e22 not found: ID does not exist" Jul 10 20:50:15.997333 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.997300 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:15.999297 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:15.999271 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74ff47775-xnvll"] Jul 10 20:50:17.985477 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:17.985438 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f342604a-3cb4-4691-a8d5-e78344e0235e" path="/var/lib/kubelet/pods/f342604a-3cb4-4691-a8d5-e78344e0235e/volumes" Jul 10 20:50:37.174207 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.174159 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5846b788db-zqdvz"] Jul 10 20:50:37.174799 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.174698 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f342604a-3cb4-4691-a8d5-e78344e0235e" containerName="manager" Jul 10 20:50:37.174799 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.174720 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f342604a-3cb4-4691-a8d5-e78344e0235e" containerName="manager" Jul 10 20:50:37.174919 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.174829 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f342604a-3cb4-4691-a8d5-e78344e0235e" containerName="manager" Jul 10 20:50:37.179703 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.179680 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.181733 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.181709 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7dl87\"" Jul 10 20:50:37.181833 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.181744 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 20:50:37.188756 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.188734 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5846b788db-zqdvz"] Jul 10 20:50:37.364436 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.364404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-openshift-service-ca\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.364603 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.364443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4gg\" (UniqueName: \"kubernetes.io/projected/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-kube-api-access-wl4gg\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.364603 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.364528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-maas-api-tls\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.465486 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.465448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-maas-api-tls\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.465674 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.465513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-openshift-service-ca\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.465674 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.465534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4gg\" (UniqueName: \"kubernetes.io/projected/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-kube-api-access-wl4gg\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.466198 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.466138 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-openshift-service-ca\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.468121 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.468098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-maas-api-tls\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.473453 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.473433 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4gg\" (UniqueName: \"kubernetes.io/projected/e8714a9e-1dc5-46bc-9ed3-a6c808ce7426-kube-api-access-wl4gg\") pod \"maas-api-5846b788db-zqdvz\" (UID: \"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426\") " pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.490412 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.490390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:37.639094 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.639066 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5846b788db-zqdvz"] Jul 10 20:50:37.641709 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:37.641678 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8714a9e_1dc5_46bc_9ed3_a6c808ce7426.slice/crio-fec7532f6825676ec4e0fc36bc9200517c20b4c8f0a5f8fd3c9e02298609a59c WatchSource:0}: Error finding container fec7532f6825676ec4e0fc36bc9200517c20b4c8f0a5f8fd3c9e02298609a59c: Status 404 returned error can't find the container with id fec7532f6825676ec4e0fc36bc9200517c20b4c8f0a5f8fd3c9e02298609a59c Jul 10 20:50:37.796380 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.796299 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dpg96"] Jul 10 20:50:37.808614 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.808587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:37.809370 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.809323 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dpg96"] Jul 10 20:50:37.810872 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.810582 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 20:50:37.810872 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.810599 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mqkbs\"" Jul 10 20:50:37.815965 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.815941 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-px95v"] Jul 10 20:50:37.819431 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.819408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:37.830628 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.830602 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-px95v"] Jul 10 20:50:37.970042 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.970002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:37.970197 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.970081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfjp\" (UniqueName: \"kubernetes.io/projected/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-kube-api-access-2pfjp\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:37.970197 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.970104 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qws4b\" (UniqueName: \"kubernetes.io/projected/0665b549-3067-4ef6-95fc-30d32cf32915-kube-api-access-qws4b\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:37.970197 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:37.970140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0665b549-3067-4ef6-95fc-30d32cf32915-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.070214 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.070115 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" event={"ID":"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426","Type":"ContainerStarted","Data":"fec7532f6825676ec4e0fc36bc9200517c20b4c8f0a5f8fd3c9e02298609a59c"} Jul 10 20:50:38.071548 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.071530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfjp\" (UniqueName: \"kubernetes.io/projected/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-kube-api-access-2pfjp\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:38.071616 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.071560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qws4b\" (UniqueName: \"kubernetes.io/projected/0665b549-3067-4ef6-95fc-30d32cf32915-kube-api-access-qws4b\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.071616 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.071590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0665b549-3067-4ef6-95fc-30d32cf32915-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.071725 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.071625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:38.072543 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.072520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0665b549-3067-4ef6-95fc-30d32cf32915-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.072619 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.072520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:38.081664 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.081633 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfjp\" (UniqueName: \"kubernetes.io/projected/deddbc3e-1c04-4cd5-b218-8aa8a445b09a-kube-api-access-2pfjp\") pod \"payload-processing-f9b8d9b9b-px95v\" (UID: \"deddbc3e-1c04-4cd5-b218-8aa8a445b09a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:38.081664 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.081650 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qws4b\" (UniqueName: \"kubernetes.io/projected/0665b549-3067-4ef6-95fc-30d32cf32915-kube-api-access-qws4b\") pod \"payload-pre-processing-74584bb9f8-dpg96\" (UID: \"0665b549-3067-4ef6-95fc-30d32cf32915\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.120854 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.120817 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:38.129112 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.129089 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:38.287073 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.286845 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dpg96"] Jul 10 20:50:38.289593 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:38.289546 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0665b549_3067_4ef6_95fc_30d32cf32915.slice/crio-21a614f7a9705551809c0ded5fee13d3370a672675fe1fa3fa5cd7c874fb9fc8 WatchSource:0}: Error finding container 21a614f7a9705551809c0ded5fee13d3370a672675fe1fa3fa5cd7c874fb9fc8: Status 404 returned error can't find the container with id 21a614f7a9705551809c0ded5fee13d3370a672675fe1fa3fa5cd7c874fb9fc8 Jul 10 20:50:38.315241 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:38.315206 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-px95v"] Jul 10 20:50:38.316827 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:38.316795 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeddbc3e_1c04_4cd5_b218_8aa8a445b09a.slice/crio-43d68c4c107ab4ccfd4811874acaa36381f86dbf1a8458ee943bab8eb6052e2f WatchSource:0}: Error finding container 43d68c4c107ab4ccfd4811874acaa36381f86dbf1a8458ee943bab8eb6052e2f: Status 404 returned error can't find the container with id 43d68c4c107ab4ccfd4811874acaa36381f86dbf1a8458ee943bab8eb6052e2f Jul 10 20:50:39.079639 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:39.079597 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" event={"ID":"deddbc3e-1c04-4cd5-b218-8aa8a445b09a","Type":"ContainerStarted","Data":"43d68c4c107ab4ccfd4811874acaa36381f86dbf1a8458ee943bab8eb6052e2f"} Jul 10 20:50:39.081363 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:39.081063 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" event={"ID":"0665b549-3067-4ef6-95fc-30d32cf32915","Type":"ContainerStarted","Data":"21a614f7a9705551809c0ded5fee13d3370a672675fe1fa3fa5cd7c874fb9fc8"} Jul 10 20:50:40.087037 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.086935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" event={"ID":"e8714a9e-1dc5-46bc-9ed3-a6c808ce7426","Type":"ContainerStarted","Data":"fe1496a141e52e41c3af7ea1e02a5349b503fcfc406c9620938965f777be2a55"} Jul 10 20:50:40.087483 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.087039 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:40.088515 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.088490 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" event={"ID":"deddbc3e-1c04-4cd5-b218-8aa8a445b09a","Type":"ContainerStarted","Data":"30bc303289a06931dd46bff42f6949959424e946e9ce6656f4c0121f915c7e61"} Jul 10 20:50:40.088607 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.088547 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:40.089950 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.089928 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" event={"ID":"0665b549-3067-4ef6-95fc-30d32cf32915","Type":"ContainerStarted","Data":"e282f0be4737acdd4c4d0174df824cfaf43e1c33e6cad831000f3af41adb0108"} Jul 10 20:50:40.090072 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.090056 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:40.103119 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.103074 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" podStartSLOduration=1.393646091 podStartE2EDuration="3.103056869s" totalImagesPullingTime="1.709410778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:40.101280089 +0000 UTC m=+770.708036693" watchObservedRunningTime="2026-07-10 20:50:40.103056869 +0000 UTC m=+770.709813451" Jul 10 20:50:40.116268 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.116193 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" podStartSLOduration=1.5913590690000001 podStartE2EDuration="3.116178329s" totalImagesPullingTime="1.52481926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:40.114351127 +0000 UTC m=+770.721107743" watchObservedRunningTime="2026-07-10 20:50:40.116178329 +0000 UTC m=+770.722934911" Jul 10 20:50:40.128918 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:40.128853 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" podStartSLOduration=1.6330416250000002 podStartE2EDuration="3.128833215s" totalImagesPullingTime="1.49579159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:40.128041606 +0000 UTC m=+770.734798208" watchObservedRunningTime="2026-07-10 20:50:40.128833215 +0000 UTC m=+770.735589803" Jul 10 20:50:46.100671 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:46.100636 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5846b788db-zqdvz" Jul 10 20:50:51.097780 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:51.097747 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-px95v" Jul 10 20:50:51.098275 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:51.098165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dpg96" Jul 10 20:50:55.266408 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.266372 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7"] Jul 10 20:50:55.270919 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.270901 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.272933 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.272907 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 20:50:55.273063 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.272980 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 20:50:55.273128 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.273060 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qhmhg\"" Jul 10 20:50:55.273183 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.273125 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 20:50:55.278621 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.278565 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7"] Jul 10 20:50:55.307189 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.307346 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.307346 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.307426 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307346 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.307426 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307372 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6nm\" (UniqueName: \"kubernetes.io/projected/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-kube-api-access-fl6nm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.307426 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.307407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408796 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408796 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408826 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408930 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6nm\" (UniqueName: \"kubernetes.io/projected/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-kube-api-access-fl6nm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.408981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.408967 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.409256 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.409204 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.409344 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.409310 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.409398 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.409341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.411258 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.411219 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.411556 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.411539 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.419592 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.419560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6nm\" (UniqueName: \"kubernetes.io/projected/45b1b509-cede-47f9-b55a-9ec8ceef1c0b-kube-api-access-fl6nm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7\" (UID: \"45b1b509-cede-47f9-b55a-9ec8ceef1c0b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.583083 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.582997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:50:55.720550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:55.720520 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7"] Jul 10 20:50:55.723922 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:50:55.723898 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b1b509_cede_47f9_b55a_9ec8ceef1c0b.slice/crio-54455ffbb0f131d5f095a5b84017d151d695e4f5b5eab9578cae36f42d5a179a WatchSource:0}: Error finding container 54455ffbb0f131d5f095a5b84017d151d695e4f5b5eab9578cae36f42d5a179a: Status 404 returned error can't find the container with id 54455ffbb0f131d5f095a5b84017d151d695e4f5b5eab9578cae36f42d5a179a Jul 10 20:50:56.171750 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:56.171710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" event={"ID":"45b1b509-cede-47f9-b55a-9ec8ceef1c0b","Type":"ContainerStarted","Data":"54455ffbb0f131d5f095a5b84017d151d695e4f5b5eab9578cae36f42d5a179a"} Jul 10 20:50:57.178430 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:57.178338 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" event={"ID":"45b1b509-cede-47f9-b55a-9ec8ceef1c0b","Type":"ContainerStarted","Data":"1434d06c6b1881f83b1ca18ffd50462d6088071c5c70334c4b26727a44c125f8"} Jul 10 20:50:57.194970 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:50:57.194908 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" podStartSLOduration=1.035650654 podStartE2EDuration="2.194889769s" totalImagesPullingTime="1.159239115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:50:57.194161684 +0000 UTC m=+787.800918292" watchObservedRunningTime="2026-07-10 20:50:57.194889769 +0000 UTC m=+787.801646349" Jul 10 20:51:05.583384 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:05.583341 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:51:05.583384 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:05.583395 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:51:05.596025 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:05.595996 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:51:06.231028 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:06.230999 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7" Jul 10 20:51:11.048342 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.048268 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf"] Jul 10 20:51:11.053981 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.053960 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.056011 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.055983 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 20:51:11.062476 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.062453 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf"] Jul 10 20:51:11.151074 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151036 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.151290 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.151290 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmts\" (UniqueName: \"kubernetes.io/projected/f848b66c-b3ed-48d7-800c-798a1a947837-kube-api-access-zvmts\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.151290 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151157 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.151468 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.151468 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.151424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f848b66c-b3ed-48d7-800c-798a1a947837-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252526 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252532 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmts\" (UniqueName: \"kubernetes.io/projected/f848b66c-b3ed-48d7-800c-798a1a947837-kube-api-access-zvmts\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.252776 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f848b66c-b3ed-48d7-800c-798a1a947837-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.253000 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.252970 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.253060 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.253011 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.253199 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.253178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.254945 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.254925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f848b66c-b3ed-48d7-800c-798a1a947837-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.255303 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.255286 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f848b66c-b3ed-48d7-800c-798a1a947837-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.259667 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.259644 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmts\" (UniqueName: \"kubernetes.io/projected/f848b66c-b3ed-48d7-800c-798a1a947837-kube-api-access-zvmts\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7txpf\" (UID: \"f848b66c-b3ed-48d7-800c-798a1a947837\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.364867 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.364764 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:11.501009 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:11.500972 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf"] Jul 10 20:51:11.501968 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:51:11.501935 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf848b66c_b3ed_48d7_800c_798a1a947837.slice/crio-4ace62e42dc1c1f4e5112e4c5f75166d56728d37e94936ca54b7f9fdb3e00dea WatchSource:0}: Error finding container 4ace62e42dc1c1f4e5112e4c5f75166d56728d37e94936ca54b7f9fdb3e00dea: Status 404 returned error can't find the container with id 4ace62e42dc1c1f4e5112e4c5f75166d56728d37e94936ca54b7f9fdb3e00dea Jul 10 20:51:12.249771 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:12.249737 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" event={"ID":"f848b66c-b3ed-48d7-800c-798a1a947837","Type":"ContainerStarted","Data":"8073fc3e424d84f901b3dbec41a8cb4d63c7798e7a6bcab536751f22385a3266"} Jul 10 20:51:12.250146 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:12.249787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" event={"ID":"f848b66c-b3ed-48d7-800c-798a1a947837","Type":"ContainerStarted","Data":"4ace62e42dc1c1f4e5112e4c5f75166d56728d37e94936ca54b7f9fdb3e00dea"} Jul 10 20:51:12.266918 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:12.266855 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" podStartSLOduration=1.07837847 podStartE2EDuration="1.26683796s" totalImagesPullingTime="188.45949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:51:12.26528851 +0000 UTC m=+802.872045095" watchObservedRunningTime="2026-07-10 20:51:12.26683796 +0000 UTC m=+802.873594544" Jul 10 20:51:21.365311 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:21.365254 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:21.365724 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:21.365341 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:21.378183 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:21.378156 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:22.308918 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:22.308886 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7txpf" Jul 10 20:51:28.048295 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.048256 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v"] Jul 10 20:51:28.054153 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.054129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.056102 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.056080 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 20:51:28.060305 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.060279 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v"] Jul 10 20:51:28.203587 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.203587 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203586 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.203814 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203613 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d8a7d-5df9-4d22-9cf3-8a548183d045-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.203814 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.203814 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.203922 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.203852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5c2\" (UniqueName: \"kubernetes.io/projected/573d8a7d-5df9-4d22-9cf3-8a548183d045-kube-api-access-ft5c2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304755 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304766 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5c2\" (UniqueName: \"kubernetes.io/projected/573d8a7d-5df9-4d22-9cf3-8a548183d045-kube-api-access-ft5c2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.304973 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.304890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d8a7d-5df9-4d22-9cf3-8a548183d045-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.305176 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.305148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.305247 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.305178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.305357 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.305337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.307332 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.307309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d8a7d-5df9-4d22-9cf3-8a548183d045-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.307457 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.307441 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d8a7d-5df9-4d22-9cf3-8a548183d045-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.312503 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.312477 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5c2\" (UniqueName: \"kubernetes.io/projected/573d8a7d-5df9-4d22-9cf3-8a548183d045-kube-api-access-ft5c2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v\" (UID: \"573d8a7d-5df9-4d22-9cf3-8a548183d045\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.367489 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.367450 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:28.503740 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:28.503654 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v"] Jul 10 20:51:28.506578 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:51:28.506545 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573d8a7d_5df9_4d22_9cf3_8a548183d045.slice/crio-34ae006466395caa1287676bda2e78ff1a8db656c89b492aac116a7905537766 WatchSource:0}: Error finding container 34ae006466395caa1287676bda2e78ff1a8db656c89b492aac116a7905537766: Status 404 returned error can't find the container with id 34ae006466395caa1287676bda2e78ff1a8db656c89b492aac116a7905537766 Jul 10 20:51:29.336045 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:29.336007 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" event={"ID":"573d8a7d-5df9-4d22-9cf3-8a548183d045","Type":"ContainerStarted","Data":"94aa10c17a5bf85180969faaf1877efa9d0fa65bf57c6fbf6697c6d30f06eb84"} Jul 10 20:51:29.336471 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:29.336061 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" event={"ID":"573d8a7d-5df9-4d22-9cf3-8a548183d045","Type":"ContainerStarted","Data":"34ae006466395caa1287676bda2e78ff1a8db656c89b492aac116a7905537766"} Jul 10 20:51:29.352362 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:29.352303 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" podStartSLOduration=1.141812228 podStartE2EDuration="1.35228535s" totalImagesPullingTime="210.473122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:51:29.351093387 +0000 UTC m=+819.957849968" watchObservedRunningTime="2026-07-10 20:51:29.35228535 +0000 UTC m=+819.959041934" Jul 10 20:51:38.367968 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:38.367927 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:38.368520 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:38.368003 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:38.380412 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:38.380389 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:38.391720 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:38.391696 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v" Jul 10 20:51:57.955258 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:57.955200 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh"] Jul 10 20:51:57.959551 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:57.959110 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:57.961093 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:57.961071 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 20:51:57.967735 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:57.967709 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh"] Jul 10 20:51:58.070019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.069975 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.070019 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.070020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfxm\" (UniqueName: \"kubernetes.io/projected/8f1e55fc-8362-4419-8c58-b033df969e06-kube-api-access-pvfxm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.070323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.070114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.070323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.070179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.070323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.070288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.070323 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.070314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1e55fc-8362-4419-8c58-b033df969e06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.171828 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.171785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.171828 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.171836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1e55fc-8362-4419-8c58-b033df969e06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172075 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.171906 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172075 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.171928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfxm\" (UniqueName: \"kubernetes.io/projected/8f1e55fc-8362-4419-8c58-b033df969e06-kube-api-access-pvfxm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172075 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.172002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172075 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.172031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172418 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.172387 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.172440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.172532 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.172502 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.174360 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.174338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1e55fc-8362-4419-8c58-b033df969e06-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.174596 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.174577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1e55fc-8362-4419-8c58-b033df969e06-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.179725 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.179703 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfxm\" (UniqueName: \"kubernetes.io/projected/8f1e55fc-8362-4419-8c58-b033df969e06-kube-api-access-pvfxm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4f9rh\" (UID: \"8f1e55fc-8362-4419-8c58-b033df969e06\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.271296 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.271205 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:51:58.410762 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.410738 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh"] Jul 10 20:51:58.413026 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:51:58.412991 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1e55fc_8362_4419_8c58_b033df969e06.slice/crio-fe22f991c271b576bbb8ddead3c6bf0405f2287b1effc8c0bf5d6f8ab54c8a2f WatchSource:0}: Error finding container fe22f991c271b576bbb8ddead3c6bf0405f2287b1effc8c0bf5d6f8ab54c8a2f: Status 404 returned error can't find the container with id fe22f991c271b576bbb8ddead3c6bf0405f2287b1effc8c0bf5d6f8ab54c8a2f Jul 10 20:51:58.476124 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:58.476082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" event={"ID":"8f1e55fc-8362-4419-8c58-b033df969e06","Type":"ContainerStarted","Data":"fe22f991c271b576bbb8ddead3c6bf0405f2287b1effc8c0bf5d6f8ab54c8a2f"} Jul 10 20:51:59.482045 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:59.482006 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" event={"ID":"8f1e55fc-8362-4419-8c58-b033df969e06","Type":"ContainerStarted","Data":"c3674204638920321e16355f51be0ba26ddca19184797c43a3a64acbf12b7683"} Jul 10 20:51:59.498899 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:51:59.498845 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" podStartSLOduration=2.3113405289999998 podStartE2EDuration="2.498823559s" totalImagesPullingTime="187.48303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:51:59.497293885 +0000 UTC m=+850.104050482" watchObservedRunningTime="2026-07-10 20:51:59.498823559 +0000 UTC m=+850.105580141" Jul 10 20:52:08.272082 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:08.272021 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:52:08.272528 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:08.272132 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:52:08.284648 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:08.284622 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:52:08.538771 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:08.538688 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4f9rh" Jul 10 20:52:49.877679 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:49.877597 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:52:49.881012 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:52:49.880985 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:55:22.615384 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.615348 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:55:22.615923 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.615563 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" podUID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" containerName="manager" containerID="cri-o://028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b" gracePeriod=30 Jul 10 20:55:22.872346 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.872281 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:55:22.953650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.953612 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") pod \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " Jul 10 20:55:22.953822 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.953710 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lctct\" (UniqueName: \"kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct\") pod \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\" (UID: \"7c6ed47b-dbe3-40fa-b134-43227cbb3872\") " Jul 10 20:55:22.955968 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.955937 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7c6ed47b-dbe3-40fa-b134-43227cbb3872" (UID: "7c6ed47b-dbe3-40fa-b134-43227cbb3872"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:55:22.956072 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:22.955972 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct" (OuterVolumeSpecName: "kube-api-access-lctct") pod "7c6ed47b-dbe3-40fa-b134-43227cbb3872" (UID: "7c6ed47b-dbe3-40fa-b134-43227cbb3872"). InnerVolumeSpecName "kube-api-access-lctct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:55:23.054879 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.054847 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctct\" (UniqueName: \"kubernetes.io/projected/7c6ed47b-dbe3-40fa-b134-43227cbb3872-kube-api-access-lctct\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:55:23.054879 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.054876 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c6ed47b-dbe3-40fa-b134-43227cbb3872-webhook-cert\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 20:55:23.459373 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.459338 2585 generic.go:356] "Generic (PLEG): container finished" podID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" containerID="028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b" exitCode=0 Jul 10 20:55:23.459550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.459383 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" event={"ID":"7c6ed47b-dbe3-40fa-b134-43227cbb3872","Type":"ContainerDied","Data":"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b"} Jul 10 20:55:23.459550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.459427 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" event={"ID":"7c6ed47b-dbe3-40fa-b134-43227cbb3872","Type":"ContainerDied","Data":"f80d8b96cabc230a0b1d51ad95282dd2574c525f601a063da64189164a2df389"} Jul 10 20:55:23.459550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.459449 2585 scope.go:117] "RemoveContainer" containerID="028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b" Jul 10 20:55:23.459550 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.459385 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-k8687" Jul 10 20:55:23.468905 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.468883 2585 scope.go:117] "RemoveContainer" containerID="028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b" Jul 10 20:55:23.469193 ip-10-0-142-194 kubenswrapper[2585]: E0710 20:55:23.469165 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b\": container with ID starting with 028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b not found: ID does not exist" containerID="028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b" Jul 10 20:55:23.469263 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.469198 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b"} err="failed to get container status \"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b\": rpc error: code = NotFound desc = could not find container \"028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b\": container with ID starting with 028d425e429b37a4e8c154cdc10bf60fd4399f21349583fa920d808d1b90296b not found: ID does not exist" Jul 10 20:55:23.478674 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.478649 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:55:23.482073 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.482042 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-k8687"] Jul 10 20:55:23.985607 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:23.985568 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" path="/var/lib/kubelet/pods/7c6ed47b-dbe3-40fa-b134-43227cbb3872/volumes" Jul 10 20:55:24.256403 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.256323 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-hlvpm"] Jul 10 20:55:24.256713 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.256701 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" containerName="manager" Jul 10 20:55:24.256751 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.256716 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" containerName="manager" Jul 10 20:55:24.256784 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.256777 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6ed47b-dbe3-40fa-b134-43227cbb3872" containerName="manager" Jul 10 20:55:24.261193 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.261169 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.263477 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.263455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 20:55:24.263593 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.263460 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wsbgn\"" Jul 10 20:55:24.263593 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.263460 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 20:55:24.268379 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.268355 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-hlvpm"] Jul 10 20:55:24.368401 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.368362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf65154a-eafc-45c9-8937-0461da6e60ef-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.368605 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.368413 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vr8z\" (UniqueName: \"kubernetes.io/projected/cf65154a-eafc-45c9-8937-0461da6e60ef-kube-api-access-6vr8z\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.469160 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.469126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf65154a-eafc-45c9-8937-0461da6e60ef-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.469366 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.469179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vr8z\" (UniqueName: \"kubernetes.io/projected/cf65154a-eafc-45c9-8937-0461da6e60ef-kube-api-access-6vr8z\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.471807 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.471774 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf65154a-eafc-45c9-8937-0461da6e60ef-webhook-cert\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.476820 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.476796 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vr8z\" (UniqueName: \"kubernetes.io/projected/cf65154a-eafc-45c9-8937-0461da6e60ef-kube-api-access-6vr8z\") pod \"maas-controller-6d9b4b8b8c-hlvpm\" (UID: \"cf65154a-eafc-45c9-8937-0461da6e60ef\") " pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.574468 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.574366 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:24.712899 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.712874 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9b4b8b8c-hlvpm"] Jul 10 20:55:24.715675 ip-10-0-142-194 kubenswrapper[2585]: W0710 20:55:24.715645 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf65154a_eafc_45c9_8937_0461da6e60ef.slice/crio-f44eb6739ed90d6bf2f9e1aa3bed043638ff22229c02292f79c65bb971601bf5 WatchSource:0}: Error finding container f44eb6739ed90d6bf2f9e1aa3bed043638ff22229c02292f79c65bb971601bf5: Status 404 returned error can't find the container with id f44eb6739ed90d6bf2f9e1aa3bed043638ff22229c02292f79c65bb971601bf5 Jul 10 20:55:24.717031 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:24.717004 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:55:25.471166 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:25.471128 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" event={"ID":"cf65154a-eafc-45c9-8937-0461da6e60ef","Type":"ContainerStarted","Data":"f44eb6739ed90d6bf2f9e1aa3bed043638ff22229c02292f79c65bb971601bf5"} Jul 10 20:55:26.477746 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:26.477707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" event={"ID":"cf65154a-eafc-45c9-8937-0461da6e60ef","Type":"ContainerStarted","Data":"2dfc52e5c4944ebaa7024919c16b0427ccebb5b5ba8f66c43fb9eae95d8af649"} Jul 10 20:55:26.478154 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:26.477862 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:55:26.492796 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:26.492730 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" podStartSLOduration=1.736825186 podStartE2EDuration="2.492717438s" totalImagesPullingTime="755.892252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:55:26.491917236 +0000 UTC m=+1057.098673818" watchObservedRunningTime="2026-07-10 20:55:26.492717438 +0000 UTC m=+1057.099474020" Jul 10 20:55:37.484650 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:55:37.484616 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9b4b8b8c-hlvpm" Jul 10 20:57:49.913703 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:57:49.913617 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 20:57:49.916916 ip-10-0-142-194 kubenswrapper[2585]: I0710 20:57:49.916896 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:00:15.004962 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:15.004930 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 21:00:15.005411 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:15.005135 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" podUID="081c4038-7afb-41d1-b378-1098ee99d918" containerName="manager" containerID="cri-o://711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8" gracePeriod=10 Jul 10 21:00:17.569720 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.569679 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" podUID="081c4038-7afb-41d1-b378-1098ee99d918" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": dial tcp 10.134.0.31:8081: connect: connection refused" Jul 10 21:00:17.765569 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.765545 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 21:00:17.814053 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.814030 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q24mq\" (UniqueName: \"kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq\") pod \"081c4038-7afb-41d1-b378-1098ee99d918\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " Jul 10 21:00:17.814194 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.814091 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume\") pod \"081c4038-7afb-41d1-b378-1098ee99d918\" (UID: \"081c4038-7afb-41d1-b378-1098ee99d918\") " Jul 10 21:00:17.814499 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.814478 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "081c4038-7afb-41d1-b378-1098ee99d918" (UID: "081c4038-7afb-41d1-b378-1098ee99d918"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 21:00:17.816698 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.816681 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq" (OuterVolumeSpecName: "kube-api-access-q24mq") pod "081c4038-7afb-41d1-b378-1098ee99d918" (UID: "081c4038-7afb-41d1-b378-1098ee99d918"). InnerVolumeSpecName "kube-api-access-q24mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:00:17.852655 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.852628 2585 generic.go:356] "Generic (PLEG): container finished" podID="081c4038-7afb-41d1-b378-1098ee99d918" containerID="711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8" exitCode=0 Jul 10 21:00:17.852767 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.852689 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" Jul 10 21:00:17.852767 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.852709 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" event={"ID":"081c4038-7afb-41d1-b378-1098ee99d918","Type":"ContainerDied","Data":"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8"} Jul 10 21:00:17.852834 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.852766 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf" event={"ID":"081c4038-7afb-41d1-b378-1098ee99d918","Type":"ContainerDied","Data":"e1d0d67e442fc66ba2aa0726a5aed02b712b9dae6a781ce67db5b03087630c80"} Jul 10 21:00:17.852834 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.852788 2585 scope.go:117] "RemoveContainer" containerID="711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8" Jul 10 21:00:17.861873 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.861856 2585 scope.go:117] "RemoveContainer" containerID="711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8" Jul 10 21:00:17.862609 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:00:17.862573 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8\": container with ID starting with 711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8 not found: ID does not exist" containerID="711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8" Jul 10 21:00:17.862708 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.862612 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8"} err="failed to get container status \"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8\": rpc error: code = NotFound desc = could not find container \"711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8\": container with ID starting with 711da74c9a4225b8f15251792fd158bdf827b100f5382e10730453cecff96cc8 not found: ID does not exist" Jul 10 21:00:17.875857 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.875833 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 21:00:17.879737 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.879716 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5rttf"] Jul 10 21:00:17.915040 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.915022 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/081c4038-7afb-41d1-b378-1098ee99d918-extensions-socket-volume\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:00:17.915127 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.915042 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q24mq\" (UniqueName: \"kubernetes.io/projected/081c4038-7afb-41d1-b378-1098ee99d918-kube-api-access-q24mq\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:00:17.985282 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:00:17.985260 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="081c4038-7afb-41d1-b378-1098ee99d918" path="/var/lib/kubelet/pods/081c4038-7afb-41d1-b378-1098ee99d918/volumes" Jul 10 21:01:16.121604 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.121573 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts"] Jul 10 21:01:16.122077 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.121985 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081c4038-7afb-41d1-b378-1098ee99d918" containerName="manager" Jul 10 21:01:16.122077 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.122002 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="081c4038-7afb-41d1-b378-1098ee99d918" containerName="manager" Jul 10 21:01:16.122171 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.122082 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="081c4038-7afb-41d1-b378-1098ee99d918" containerName="manager" Jul 10 21:01:16.126279 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.126261 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.128164 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.128142 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j6t8z\"" Jul 10 21:01:16.134545 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.134522 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts"] Jul 10 21:01:16.224264 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.224189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88af342a-9612-447a-bf8e-25ca984a716d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.224451 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.224313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwtm\" (UniqueName: \"kubernetes.io/projected/88af342a-9612-447a-bf8e-25ca984a716d-kube-api-access-skwtm\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.325067 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.325033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88af342a-9612-447a-bf8e-25ca984a716d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.325277 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.325075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwtm\" (UniqueName: \"kubernetes.io/projected/88af342a-9612-447a-bf8e-25ca984a716d-kube-api-access-skwtm\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.325416 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.325396 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88af342a-9612-447a-bf8e-25ca984a716d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.333337 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.333306 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwtm\" (UniqueName: \"kubernetes.io/projected/88af342a-9612-447a-bf8e-25ca984a716d-kube-api-access-skwtm\") pod \"kuadrant-operator-controller-manager-86878c769f-xsbts\" (UID: \"88af342a-9612-447a-bf8e-25ca984a716d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.437130 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.437039 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:16.576201 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:16.575768 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts"] Jul 10 21:01:16.578449 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:01:16.578422 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af342a_9612_447a_bf8e_25ca984a716d.slice/crio-7f36a1c04b4e5d6f9cc9782d95a85921968d5428085dc05727c4dc65fe2ad1f9 WatchSource:0}: Error finding container 7f36a1c04b4e5d6f9cc9782d95a85921968d5428085dc05727c4dc65fe2ad1f9: Status 404 returned error can't find the container with id 7f36a1c04b4e5d6f9cc9782d95a85921968d5428085dc05727c4dc65fe2ad1f9 Jul 10 21:01:17.127560 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:17.127522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" event={"ID":"88af342a-9612-447a-bf8e-25ca984a716d","Type":"ContainerStarted","Data":"d760f6590f4181b479bc8883b4dfaf089683e38fcb8fa9f5275f7347b9340805"} Jul 10 21:01:17.127956 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:17.127579 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" event={"ID":"88af342a-9612-447a-bf8e-25ca984a716d","Type":"ContainerStarted","Data":"7f36a1c04b4e5d6f9cc9782d95a85921968d5428085dc05727c4dc65fe2ad1f9"} Jul 10 21:01:17.127956 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:17.127625 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:01:17.149933 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:17.149877 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" podStartSLOduration=1.149861621 podStartE2EDuration="1.149861621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:01:17.1465412 +0000 UTC m=+1407.753297797" watchObservedRunningTime="2026-07-10 21:01:17.149861621 +0000 UTC m=+1407.756618203" Jul 10 21:01:28.134776 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:01:28.134742 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xsbts" Jul 10 21:02:49.941177 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:02:49.941144 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:02:49.945365 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:02:49.945340 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:07:32.148977 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.148943 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:07:32.152181 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.152156 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.154046 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.154026 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-35794783-serving-cert\"" Jul 10 21:07:32.163644 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.163621 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:07:32.291107 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.291075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.291318 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.291192 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9nm\" (UniqueName: \"kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.291372 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.291314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.392225 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.392189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9nm\" (UniqueName: \"kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.392454 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.392268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.392454 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.392308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.392965 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.392939 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.395203 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.395175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.400614 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.400550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9nm\" (UniqueName: \"kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm\") pod \"maas-api-e2e-ait-35794783-86579c4689-c7v5d\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.462907 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.462875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:32.600622 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.600592 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:07:32.602564 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:07:32.602539 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d95d88_8cf7_4f1a_9e7d_0e1ad9329989.slice/crio-4437f799f83c75664879f37c87f86581fd2c9ae637f586d41e4b16e056fc23b9 WatchSource:0}: Error finding container 4437f799f83c75664879f37c87f86581fd2c9ae637f586d41e4b16e056fc23b9: Status 404 returned error can't find the container with id 4437f799f83c75664879f37c87f86581fd2c9ae637f586d41e4b16e056fc23b9 Jul 10 21:07:32.604483 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.604456 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:07:32.905072 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:32.905040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" event={"ID":"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989","Type":"ContainerStarted","Data":"4437f799f83c75664879f37c87f86581fd2c9ae637f586d41e4b16e056fc23b9"} Jul 10 21:07:33.911975 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:33.911941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" event={"ID":"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989","Type":"ContainerStarted","Data":"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57"} Jul 10 21:07:33.912387 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:33.912063 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:33.927764 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:33.927713 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" podStartSLOduration=1.427087599 podStartE2EDuration="1.92769698s" totalImagesPullingTime="500.609381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:07:33.926445338 +0000 UTC m=+1784.533201941" watchObservedRunningTime="2026-07-10 21:07:33.92769698 +0000 UTC m=+1784.534453563" Jul 10 21:07:39.923631 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:39.923552 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:07:49.968679 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:49.968652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:07:49.974762 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:07:49.974742 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:08:37.694913 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.694882 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:08:37.695566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.695134 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" podUID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" containerName="maas-api" containerID="cri-o://85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57" gracePeriod=30 Jul 10 21:08:37.953478 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.953413 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:08:37.974787 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.974761 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn9nm\" (UniqueName: \"kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm\") pod \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " Jul 10 21:08:37.974907 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.974800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca\") pod \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " Jul 10 21:08:37.974907 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.974881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls\") pod \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\" (UID: \"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989\") " Jul 10 21:08:37.975197 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.975176 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" (UID: "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:08:37.977083 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.977054 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" (UID: "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:08:37.977166 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:37.977096 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm" (OuterVolumeSpecName: "kube-api-access-xn9nm") pod "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" (UID: "a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989"). InnerVolumeSpecName "kube-api-access-xn9nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:08:38.075740 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.075717 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:08:38.075740 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.075740 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9nm\" (UniqueName: \"kubernetes.io/projected/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-kube-api-access-xn9nm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:08:38.075870 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.075751 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:08:38.205529 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.205449 2585 generic.go:356] "Generic (PLEG): container finished" podID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" containerID="85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57" exitCode=0 Jul 10 21:08:38.205529 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.205499 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" Jul 10 21:08:38.205529 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.205526 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" event={"ID":"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989","Type":"ContainerDied","Data":"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57"} Jul 10 21:08:38.205749 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.205569 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d" event={"ID":"a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989","Type":"ContainerDied","Data":"4437f799f83c75664879f37c87f86581fd2c9ae637f586d41e4b16e056fc23b9"} Jul 10 21:08:38.205749 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.205591 2585 scope.go:117] "RemoveContainer" containerID="85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57" Jul 10 21:08:38.214291 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.214280 2585 scope.go:117] "RemoveContainer" containerID="85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57" Jul 10 21:08:38.214559 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:08:38.214537 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57\": container with ID starting with 85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57 not found: ID does not exist" containerID="85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57" Jul 10 21:08:38.214606 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.214561 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57"} err="failed to get container status \"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57\": rpc error: code = NotFound desc = could not find container \"85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57\": container with ID starting with 85079fe00cb0bea03a1d0d29e694d23d1aa2e93706af138492662cb89b980c57 not found: ID does not exist" Jul 10 21:08:38.230017 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.227842 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:08:38.234151 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:38.233900 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35794783-86579c4689-c7v5d"] Jul 10 21:08:39.985790 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:39.985753 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" path="/var/lib/kubelet/pods/a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989/volumes" Jul 10 21:08:54.759948 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.759912 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:08:54.760514 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.760498 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" containerName="maas-api" Jul 10 21:08:54.760559 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.760518 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" containerName="maas-api" Jul 10 21:08:54.760637 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.760624 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d95d88-8cf7-4f1a-9e7d-0e1ad9329989" containerName="maas-api" Jul 10 21:08:54.763933 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.763911 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.765902 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.765877 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6fde404a-serving-cert\"" Jul 10 21:08:54.772132 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.772110 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:08:54.824767 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.824731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgp5\" (UniqueName: \"kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.824895 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.824792 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.824895 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.824833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.925821 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.925785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgp5\" (UniqueName: \"kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.925980 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.925847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.925980 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.925893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.926696 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.926675 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.928501 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.928483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:54.933426 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:54.933410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgp5\" (UniqueName: \"kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5\") pod \"maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:55.076220 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:55.076132 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:55.213262 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:55.213214 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:08:55.216807 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:08:55.216775 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c24d7c0_973f_4263_81b1_bf8e0ae8449a.slice/crio-7670a9cf59be0cc7c8c32ace260b87586d1e2a07c12585f14db637cc1da8ff7d WatchSource:0}: Error finding container 7670a9cf59be0cc7c8c32ace260b87586d1e2a07c12585f14db637cc1da8ff7d: Status 404 returned error can't find the container with id 7670a9cf59be0cc7c8c32ace260b87586d1e2a07c12585f14db637cc1da8ff7d Jul 10 21:08:55.289648 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:55.289617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" event={"ID":"0c24d7c0-973f-4263-81b1-bf8e0ae8449a","Type":"ContainerStarted","Data":"7670a9cf59be0cc7c8c32ace260b87586d1e2a07c12585f14db637cc1da8ff7d"} Jul 10 21:08:56.296420 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:56.296381 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" event={"ID":"0c24d7c0-973f-4263-81b1-bf8e0ae8449a","Type":"ContainerStarted","Data":"dab4b53166267c34fbfae7fc4dca7eaa9b64e86082d49c2f4e7a6ca58f139a40"} Jul 10 21:08:56.296837 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:56.296517 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:08:56.315775 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:08:56.315730 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" podStartSLOduration=1.818117312 podStartE2EDuration="2.315712292s" totalImagesPullingTime="497.59498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:08:56.314887693 +0000 UTC m=+1866.921644274" watchObservedRunningTime="2026-07-10 21:08:56.315712292 +0000 UTC m=+1866.922468878" Jul 10 21:09:02.308156 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:02.308121 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:09:59.409745 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.409710 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:09:59.410356 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.410031 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" podUID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" containerName="maas-api" containerID="cri-o://dab4b53166267c34fbfae7fc4dca7eaa9b64e86082d49c2f4e7a6ca58f139a40" gracePeriod=30 Jul 10 21:09:59.602063 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.601937 2585 generic.go:356] "Generic (PLEG): container finished" podID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" containerID="dab4b53166267c34fbfae7fc4dca7eaa9b64e86082d49c2f4e7a6ca58f139a40" exitCode=0 Jul 10 21:09:59.602063 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.602034 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" event={"ID":"0c24d7c0-973f-4263-81b1-bf8e0ae8449a","Type":"ContainerDied","Data":"dab4b53166267c34fbfae7fc4dca7eaa9b64e86082d49c2f4e7a6ca58f139a40"} Jul 10 21:09:59.659368 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.659341 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:09:59.792152 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.792119 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls\") pod \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " Jul 10 21:09:59.792362 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.792213 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca\") pod \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " Jul 10 21:09:59.792362 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.792273 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mgp5\" (UniqueName: \"kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5\") pod \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\" (UID: \"0c24d7c0-973f-4263-81b1-bf8e0ae8449a\") " Jul 10 21:09:59.792578 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.792553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c24d7c0-973f-4263-81b1-bf8e0ae8449a" (UID: "0c24d7c0-973f-4263-81b1-bf8e0ae8449a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:09:59.794790 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.794756 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c24d7c0-973f-4263-81b1-bf8e0ae8449a" (UID: "0c24d7c0-973f-4263-81b1-bf8e0ae8449a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:09:59.794918 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.794881 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5" (OuterVolumeSpecName: "kube-api-access-7mgp5") pod "0c24d7c0-973f-4263-81b1-bf8e0ae8449a" (UID: "0c24d7c0-973f-4263-81b1-bf8e0ae8449a"). InnerVolumeSpecName "kube-api-access-7mgp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:09:59.893063 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.893030 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:09:59.893063 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.893056 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgp5\" (UniqueName: \"kubernetes.io/projected/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-kube-api-access-7mgp5\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:09:59.893063 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:09:59.893068 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c24d7c0-973f-4263-81b1-bf8e0ae8449a-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:10:00.608383 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:00.608354 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" Jul 10 21:10:00.608759 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:00.608354 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt" event={"ID":"0c24d7c0-973f-4263-81b1-bf8e0ae8449a","Type":"ContainerDied","Data":"7670a9cf59be0cc7c8c32ace260b87586d1e2a07c12585f14db637cc1da8ff7d"} Jul 10 21:10:00.608759 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:00.608490 2585 scope.go:117] "RemoveContainer" containerID="dab4b53166267c34fbfae7fc4dca7eaa9b64e86082d49c2f4e7a6ca58f139a40" Jul 10 21:10:00.623849 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:00.623825 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:10:00.627087 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:00.627065 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6fde404a-6bdb7c774d-dq2qt"] Jul 10 21:10:01.985014 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:01.984976 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" path="/var/lib/kubelet/pods/0c24d7c0-973f-4263-81b1-bf8e0ae8449a/volumes" Jul 10 21:10:06.460378 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.460339 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:10:06.460760 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.460733 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" containerName="maas-api" Jul 10 21:10:06.460760 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.460744 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" containerName="maas-api" Jul 10 21:10:06.460872 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.460825 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c24d7c0-973f-4263-81b1-bf8e0ae8449a" containerName="maas-api" Jul 10 21:10:06.465171 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.465149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.467026 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.466988 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6bb25bb7-serving-cert\"" Jul 10 21:10:06.475688 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.475665 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:10:06.549103 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.549063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf556\" (UniqueName: \"kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.549103 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.549100 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.549333 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.549119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.650416 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.650385 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf556\" (UniqueName: \"kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.650416 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.650421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.650623 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.650439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.651328 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.651303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.652969 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.652951 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.658039 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.658017 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf556\" (UniqueName: \"kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556\") pod \"maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.776756 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.776670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:06.913812 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:06.913788 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:10:06.915803 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:10:06.915777 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff9a5c9_762d_42c7_8b96_d1bf6dd58392.slice/crio-c07ccabe272423565ec7827e652028faedaa20d33cc5e9763a53140dd12ab2d3 WatchSource:0}: Error finding container c07ccabe272423565ec7827e652028faedaa20d33cc5e9763a53140dd12ab2d3: Status 404 returned error can't find the container with id c07ccabe272423565ec7827e652028faedaa20d33cc5e9763a53140dd12ab2d3 Jul 10 21:10:07.643315 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:07.643269 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" event={"ID":"aff9a5c9-762d-42c7-8b96-d1bf6dd58392","Type":"ContainerStarted","Data":"68d4122ecea4e78dbfcabce255a974b93c47e2d873f164ff1a20d751d740e323"} Jul 10 21:10:07.643699 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:07.643326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" event={"ID":"aff9a5c9-762d-42c7-8b96-d1bf6dd58392","Type":"ContainerStarted","Data":"c07ccabe272423565ec7827e652028faedaa20d33cc5e9763a53140dd12ab2d3"} Jul 10 21:10:07.643699 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:07.643349 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:10:07.660048 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:07.659998 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" podStartSLOduration=1.070579936 podStartE2EDuration="1.659977499s" totalImagesPullingTime="589.397563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:10:07.657802532 +0000 UTC m=+1938.264559114" watchObservedRunningTime="2026-07-10 21:10:07.659977499 +0000 UTC m=+1938.266734082" Jul 10 21:10:13.654191 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:10:13.654160 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:11:21.803571 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:21.803536 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:11:21.804173 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:21.803754 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" podUID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" containerName="maas-api" containerID="cri-o://68d4122ecea4e78dbfcabce255a974b93c47e2d873f164ff1a20d751d740e323" gracePeriod=30 Jul 10 21:11:21.995459 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:21.995295 2585 generic.go:356] "Generic (PLEG): container finished" podID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" containerID="68d4122ecea4e78dbfcabce255a974b93c47e2d873f164ff1a20d751d740e323" exitCode=0 Jul 10 21:11:21.995459 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:21.995373 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" event={"ID":"aff9a5c9-762d-42c7-8b96-d1bf6dd58392","Type":"ContainerDied","Data":"68d4122ecea4e78dbfcabce255a974b93c47e2d873f164ff1a20d751d740e323"} Jul 10 21:11:22.060013 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.059956 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:11:22.102583 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.102552 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf556\" (UniqueName: \"kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556\") pod \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " Jul 10 21:11:22.102752 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.102601 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca\") pod \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " Jul 10 21:11:22.102752 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.102735 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls\") pod \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\" (UID: \"aff9a5c9-762d-42c7-8b96-d1bf6dd58392\") " Jul 10 21:11:22.103014 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.102985 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aff9a5c9-762d-42c7-8b96-d1bf6dd58392" (UID: "aff9a5c9-762d-42c7-8b96-d1bf6dd58392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:11:22.103146 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.103086 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:11:22.104951 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.104921 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556" (OuterVolumeSpecName: "kube-api-access-lf556") pod "aff9a5c9-762d-42c7-8b96-d1bf6dd58392" (UID: "aff9a5c9-762d-42c7-8b96-d1bf6dd58392"). InnerVolumeSpecName "kube-api-access-lf556". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:11:22.105053 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.104983 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aff9a5c9-762d-42c7-8b96-d1bf6dd58392" (UID: "aff9a5c9-762d-42c7-8b96-d1bf6dd58392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:11:22.204470 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.204440 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:11:22.204470 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:22.204467 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf556\" (UniqueName: \"kubernetes.io/projected/aff9a5c9-762d-42c7-8b96-d1bf6dd58392-kube-api-access-lf556\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:11:23.002204 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.002177 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" Jul 10 21:11:23.002617 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.002167 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs" event={"ID":"aff9a5c9-762d-42c7-8b96-d1bf6dd58392","Type":"ContainerDied","Data":"c07ccabe272423565ec7827e652028faedaa20d33cc5e9763a53140dd12ab2d3"} Jul 10 21:11:23.002617 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.002321 2585 scope.go:117] "RemoveContainer" containerID="68d4122ecea4e78dbfcabce255a974b93c47e2d873f164ff1a20d751d740e323" Jul 10 21:11:23.022448 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.022400 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:11:23.029964 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.025762 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6bb25bb7-765b8bd64f-lc4xs"] Jul 10 21:11:23.985144 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:23.985108 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" path="/var/lib/kubelet/pods/aff9a5c9-762d-42c7-8b96-d1bf6dd58392/volumes" Jul 10 21:11:38.942946 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.942893 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:11:38.943569 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.943545 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" containerName="maas-api" Jul 10 21:11:38.943750 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.943728 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" containerName="maas-api" Jul 10 21:11:38.943920 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.943901 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff9a5c9-762d-42c7-8b96-d1bf6dd58392" containerName="maas-api" Jul 10 21:11:38.948068 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.948051 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:38.950156 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.950128 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-183404a5-serving-cert\"" Jul 10 21:11:38.954042 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:38.954020 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:11:39.062710 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.062674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggg4\" (UniqueName: \"kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.062874 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.062742 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.062913 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.062879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.163784 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.163748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.163945 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.163796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggg4\" (UniqueName: \"kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.163945 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.163839 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.164446 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.164426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.166412 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.166389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.172561 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.172539 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggg4\" (UniqueName: \"kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4\") pod \"maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.261279 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.261183 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:39.401624 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:39.401595 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:11:39.402411 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:11:39.402386 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec4120c_930d_4da8_b939_7fec7c958506.slice/crio-08c411aa0ad87cac451f86639defa77fc307927ef440ef78c5310b620172d3eb WatchSource:0}: Error finding container 08c411aa0ad87cac451f86639defa77fc307927ef440ef78c5310b620172d3eb: Status 404 returned error can't find the container with id 08c411aa0ad87cac451f86639defa77fc307927ef440ef78c5310b620172d3eb Jul 10 21:11:40.088268 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:40.088140 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" event={"ID":"fec4120c-930d-4da8-b939-7fec7c958506","Type":"ContainerStarted","Data":"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb"} Jul 10 21:11:40.088268 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:40.088199 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" event={"ID":"fec4120c-930d-4da8-b939-7fec7c958506","Type":"ContainerStarted","Data":"08c411aa0ad87cac451f86639defa77fc307927ef440ef78c5310b620172d3eb"} Jul 10 21:11:40.088711 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:40.088292 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:11:40.104804 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:40.104755 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" podStartSLOduration=1.705475574 podStartE2EDuration="2.104738911s" totalImagesPullingTime="399.263337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:11:40.102822799 +0000 UTC m=+2030.709579397" watchObservedRunningTime="2026-07-10 21:11:40.104738911 +0000 UTC m=+2030.711495493" Jul 10 21:11:46.105430 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:11:46.105399 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:12:44.061748 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.061712 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:12:44.062277 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.061968 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" podUID="fec4120c-930d-4da8-b939-7fec7c958506" containerName="maas-api" containerID="cri-o://412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb" gracePeriod=30 Jul 10 21:12:44.324068 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.324005 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:12:44.365300 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.365266 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca\") pod \"fec4120c-930d-4da8-b939-7fec7c958506\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " Jul 10 21:12:44.365469 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.365340 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wggg4\" (UniqueName: \"kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4\") pod \"fec4120c-930d-4da8-b939-7fec7c958506\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " Jul 10 21:12:44.365469 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.365380 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls\") pod \"fec4120c-930d-4da8-b939-7fec7c958506\" (UID: \"fec4120c-930d-4da8-b939-7fec7c958506\") " Jul 10 21:12:44.365674 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.365650 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fec4120c-930d-4da8-b939-7fec7c958506" (UID: "fec4120c-930d-4da8-b939-7fec7c958506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:12:44.367621 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.367585 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fec4120c-930d-4da8-b939-7fec7c958506" (UID: "fec4120c-930d-4da8-b939-7fec7c958506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:12:44.367714 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.367644 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4" (OuterVolumeSpecName: "kube-api-access-wggg4") pod "fec4120c-930d-4da8-b939-7fec7c958506" (UID: "fec4120c-930d-4da8-b939-7fec7c958506"). InnerVolumeSpecName "kube-api-access-wggg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:12:44.406626 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.406594 2585 generic.go:356] "Generic (PLEG): container finished" podID="fec4120c-930d-4da8-b939-7fec7c958506" containerID="412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb" exitCode=0 Jul 10 21:12:44.406813 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.406632 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" event={"ID":"fec4120c-930d-4da8-b939-7fec7c958506","Type":"ContainerDied","Data":"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb"} Jul 10 21:12:44.406813 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.406653 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" Jul 10 21:12:44.406813 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.406669 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z" event={"ID":"fec4120c-930d-4da8-b939-7fec7c958506","Type":"ContainerDied","Data":"08c411aa0ad87cac451f86639defa77fc307927ef440ef78c5310b620172d3eb"} Jul 10 21:12:44.406813 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.406691 2585 scope.go:117] "RemoveContainer" containerID="412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb" Jul 10 21:12:44.415663 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.415645 2585 scope.go:117] "RemoveContainer" containerID="412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb" Jul 10 21:12:44.415946 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:12:44.415926 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb\": container with ID starting with 412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb not found: ID does not exist" containerID="412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb" Jul 10 21:12:44.415993 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.415958 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb"} err="failed to get container status \"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb\": rpc error: code = NotFound desc = could not find container \"412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb\": container with ID starting with 412b9643b64cb3ba78743273552e57b61c36c8a95c23f2f3ee18b0d02f8e1bdb not found: ID does not exist" Jul 10 21:12:44.433508 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.433483 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:12:44.435838 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.435815 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-183404a5-7fdd679c4-t2s6z"] Jul 10 21:12:44.466346 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.466315 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec4120c-930d-4da8-b939-7fec7c958506-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:12:44.466346 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.466341 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec4120c-930d-4da8-b939-7fec7c958506-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:12:44.466346 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:44.466352 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggg4\" (UniqueName: \"kubernetes.io/projected/fec4120c-930d-4da8-b939-7fec7c958506-kube-api-access-wggg4\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:12:45.985384 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:45.985351 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec4120c-930d-4da8-b939-7fec7c958506" path="/var/lib/kubelet/pods/fec4120c-930d-4da8-b939-7fec7c958506/volumes" Jul 10 21:12:49.996400 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:49.996373 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:12:50.004490 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:12:50.004464 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:14:38.924616 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.924587 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:14:38.925078 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.924968 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec4120c-930d-4da8-b939-7fec7c958506" containerName="maas-api" Jul 10 21:14:38.925078 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.924979 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec4120c-930d-4da8-b939-7fec7c958506" containerName="maas-api" Jul 10 21:14:38.925078 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.925043 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec4120c-930d-4da8-b939-7fec7c958506" containerName="maas-api" Jul 10 21:14:38.928299 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.928277 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:38.930486 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.930459 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd381648-serving-cert\"" Jul 10 21:14:38.940631 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:38.940598 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:14:39.108661 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.108617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4gp\" (UniqueName: \"kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.108864 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.108666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.108864 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.108741 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.210181 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.210144 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4gp\" (UniqueName: \"kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.210412 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.210186 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.210412 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.210270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.210412 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:14:39.210377 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-serving-cert: secret "maas-api-e2e-mt-dd381648-serving-cert" not found Jul 10 21:14:39.210593 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:14:39.210458 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls podName:c9b2a9cc-41a3-44fc-b498-77910dc3ffb4 nodeName:}" failed. No retries permitted until 2026-07-10 21:14:39.710438539 +0000 UTC m=+2210.317195098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls") pod "maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" (UID: "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4") : secret "maas-api-e2e-mt-dd381648-serving-cert" not found Jul 10 21:14:39.210839 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.210822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.228733 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.228695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4gp\" (UniqueName: \"kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.714960 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.714924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.717524 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.717504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") pod \"maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.842698 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.842653 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:39.989275 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.989196 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:14:39.991833 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:14:39.991806 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b2a9cc_41a3_44fc_b498_77910dc3ffb4.slice/crio-e5a481f9eeb40542ac7d5945129dcac69b9ab86793c70faffdee40663f403c65 WatchSource:0}: Error finding container e5a481f9eeb40542ac7d5945129dcac69b9ab86793c70faffdee40663f403c65: Status 404 returned error can't find the container with id e5a481f9eeb40542ac7d5945129dcac69b9ab86793c70faffdee40663f403c65 Jul 10 21:14:39.993204 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:39.993187 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:14:40.968097 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:40.968057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" event={"ID":"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4","Type":"ContainerStarted","Data":"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14"} Jul 10 21:14:40.968304 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:40.968145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" event={"ID":"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4","Type":"ContainerStarted","Data":"e5a481f9eeb40542ac7d5945129dcac69b9ab86793c70faffdee40663f403c65"} Jul 10 21:14:40.968304 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:40.968206 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:14:40.985819 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:40.985763 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" podStartSLOduration=2.396418155 podStartE2EDuration="2.985744448s" totalImagesPullingTime="589.326293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:14:40.984669308 +0000 UTC m=+2211.591425931" watchObservedRunningTime="2026-07-10 21:14:40.985744448 +0000 UTC m=+2211.592501030" Jul 10 21:14:46.978652 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:14:46.978623 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:15:01.665448 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:01.665414 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:15:01.665906 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:01.665610 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" podUID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" containerName="maas-api" containerID="cri-o://8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14" gracePeriod=30 Jul 10 21:15:01.913481 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:01.913454 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:15:02.004752 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.004723 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") pod \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " Jul 10 21:15:02.004919 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.004774 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca\") pod \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " Jul 10 21:15:02.004919 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.004837 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht4gp\" (UniqueName: \"kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp\") pod \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\" (UID: \"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4\") " Jul 10 21:15:02.005255 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.005196 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" (UID: "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:15:02.007033 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.006997 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" (UID: "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:15:02.007121 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.007073 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp" (OuterVolumeSpecName: "kube-api-access-ht4gp") pod "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" (UID: "c9b2a9cc-41a3-44fc-b498-77910dc3ffb4"). InnerVolumeSpecName "kube-api-access-ht4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:15:02.066919 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.066887 2585 generic.go:356] "Generic (PLEG): container finished" podID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" containerID="8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14" exitCode=0 Jul 10 21:15:02.067049 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.066928 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" Jul 10 21:15:02.067049 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.066934 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" event={"ID":"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4","Type":"ContainerDied","Data":"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14"} Jul 10 21:15:02.067049 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.066976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64" event={"ID":"c9b2a9cc-41a3-44fc-b498-77910dc3ffb4","Type":"ContainerDied","Data":"e5a481f9eeb40542ac7d5945129dcac69b9ab86793c70faffdee40663f403c65"} Jul 10 21:15:02.067049 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.066999 2585 scope.go:117] "RemoveContainer" containerID="8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14" Jul 10 21:15:02.075683 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.075665 2585 scope.go:117] "RemoveContainer" containerID="8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14" Jul 10 21:15:02.075940 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:15:02.075921 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14\": container with ID starting with 8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14 not found: ID does not exist" containerID="8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14" Jul 10 21:15:02.075977 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.075950 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14"} err="failed to get container status \"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14\": rpc error: code = NotFound desc = could not find container \"8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14\": container with ID starting with 8643f5ded131edbedfeddb76fa3707a4f08e8e19bbf5581b83281f0b72116c14 not found: ID does not exist" Jul 10 21:15:02.090900 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.090874 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:15:02.098064 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.098042 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd381648-5fc5c4ff8d-27q64"] Jul 10 21:15:02.105935 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.105912 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:15:02.105935 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.105932 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:15:02.106100 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:02.105942 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4gp\" (UniqueName: \"kubernetes.io/projected/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4-kube-api-access-ht4gp\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:15:03.985143 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:03.985109 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" path="/var/lib/kubelet/pods/c9b2a9cc-41a3-44fc-b498-77910dc3ffb4/volumes" Jul 10 21:15:40.288835 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.288801 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:15:40.289298 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.289200 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" containerName="maas-api" Jul 10 21:15:40.289298 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.289211 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" containerName="maas-api" Jul 10 21:15:40.289298 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.289289 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b2a9cc-41a3-44fc-b498-77910dc3ffb4" containerName="maas-api" Jul 10 21:15:40.292056 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.292034 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.294024 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.294004 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-46c4a127-serving-cert\"" Jul 10 21:15:40.302121 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.302092 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:15:40.358345 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.358301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.358539 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.358469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrpr\" (UniqueName: \"kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.358539 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.358510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.459854 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.459822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrpr\" (UniqueName: \"kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.460177 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.459871 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.460177 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.459900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.460177 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:15:40.459986 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-serving-cert: secret "maas-api-e2e-mt-46c4a127-serving-cert" not found Jul 10 21:15:40.460177 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:15:40.460063 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls podName:476b7d40-bcc1-41ba-80e2-a0d29a94b12f nodeName:}" failed. No retries permitted until 2026-07-10 21:15:40.960042644 +0000 UTC m=+2271.566799207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls") pod "maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" (UID: "476b7d40-bcc1-41ba-80e2-a0d29a94b12f") : secret "maas-api-e2e-mt-46c4a127-serving-cert" not found Jul 10 21:15:40.460632 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.460605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.470110 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.470084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrpr\" (UniqueName: \"kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.964915 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.964885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:40.967473 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:40.967453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") pod \"maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:41.204964 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:41.204927 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:41.352733 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:41.352704 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:15:41.355406 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:15:41.355378 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476b7d40_bcc1_41ba_80e2_a0d29a94b12f.slice/crio-79cbf8b95ff39f73065bbe0ee1bd197db4e8e23f5a7eb691c9db6cb6d8085a7f WatchSource:0}: Error finding container 79cbf8b95ff39f73065bbe0ee1bd197db4e8e23f5a7eb691c9db6cb6d8085a7f: Status 404 returned error can't find the container with id 79cbf8b95ff39f73065bbe0ee1bd197db4e8e23f5a7eb691c9db6cb6d8085a7f Jul 10 21:15:42.257858 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:42.257816 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" event={"ID":"476b7d40-bcc1-41ba-80e2-a0d29a94b12f","Type":"ContainerStarted","Data":"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d"} Jul 10 21:15:42.258056 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:42.257869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" event={"ID":"476b7d40-bcc1-41ba-80e2-a0d29a94b12f","Type":"ContainerStarted","Data":"79cbf8b95ff39f73065bbe0ee1bd197db4e8e23f5a7eb691c9db6cb6d8085a7f"} Jul 10 21:15:42.258056 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:42.257907 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:15:42.276932 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:42.276877 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" podStartSLOduration=1.869623169 podStartE2EDuration="2.276862042s" totalImagesPullingTime="407.238873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:15:42.274800193 +0000 UTC m=+2272.881556787" watchObservedRunningTime="2026-07-10 21:15:42.276862042 +0000 UTC m=+2272.883618624" Jul 10 21:15:48.269768 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:15:48.269735 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:16:18.134209 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.134170 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:16:18.134842 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.134433 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerName="maas-api" containerID="cri-o://144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d" gracePeriod=30 Jul 10 21:16:18.263376 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.263322 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.51:8443/health\": dial tcp 10.134.0.51:8443: connect: connection refused" Jul 10 21:16:18.395569 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.395493 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:16:18.411135 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.411097 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca\") pod \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " Jul 10 21:16:18.411366 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.411191 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlrpr\" (UniqueName: \"kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr\") pod \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " Jul 10 21:16:18.411366 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.411343 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") pod \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\" (UID: \"476b7d40-bcc1-41ba-80e2-a0d29a94b12f\") " Jul 10 21:16:18.411921 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.411810 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "476b7d40-bcc1-41ba-80e2-a0d29a94b12f" (UID: "476b7d40-bcc1-41ba-80e2-a0d29a94b12f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:16:18.414210 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.414147 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr" (OuterVolumeSpecName: "kube-api-access-nlrpr") pod "476b7d40-bcc1-41ba-80e2-a0d29a94b12f" (UID: "476b7d40-bcc1-41ba-80e2-a0d29a94b12f"). InnerVolumeSpecName "kube-api-access-nlrpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:16:18.414210 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.414159 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "476b7d40-bcc1-41ba-80e2-a0d29a94b12f" (UID: "476b7d40-bcc1-41ba-80e2-a0d29a94b12f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:16:18.435429 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.435394 2585 generic.go:356] "Generic (PLEG): container finished" podID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerID="144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d" exitCode=0 Jul 10 21:16:18.435566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.435451 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" Jul 10 21:16:18.435566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.435476 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" event={"ID":"476b7d40-bcc1-41ba-80e2-a0d29a94b12f","Type":"ContainerDied","Data":"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d"} Jul 10 21:16:18.435566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.435531 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx" event={"ID":"476b7d40-bcc1-41ba-80e2-a0d29a94b12f","Type":"ContainerDied","Data":"79cbf8b95ff39f73065bbe0ee1bd197db4e8e23f5a7eb691c9db6cb6d8085a7f"} Jul 10 21:16:18.435566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.435554 2585 scope.go:117] "RemoveContainer" containerID="144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d" Jul 10 21:16:18.444762 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.444740 2585 scope.go:117] "RemoveContainer" containerID="144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d" Jul 10 21:16:18.445046 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:16:18.445027 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d\": container with ID starting with 144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d not found: ID does not exist" containerID="144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d" Jul 10 21:16:18.445138 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.445057 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d"} err="failed to get container status \"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d\": rpc error: code = NotFound desc = could not find container \"144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d\": container with ID starting with 144eb627d9f97b06fb2fadb95b58ce3b7e6345510de3c836eed230ba40c2504d not found: ID does not exist" Jul 10 21:16:18.458824 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.458800 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:16:18.462776 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.462754 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46c4a127-7b85f5c776-gw8qx"] Jul 10 21:16:18.512507 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.512469 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:16:18.512507 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.512497 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:16:18.512507 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:18.512513 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrpr\" (UniqueName: \"kubernetes.io/projected/476b7d40-bcc1-41ba-80e2-a0d29a94b12f-kube-api-access-nlrpr\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:16:19.985822 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:16:19.985789 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" path="/var/lib/kubelet/pods/476b7d40-bcc1-41ba-80e2-a0d29a94b12f/volumes" Jul 10 21:17:50.023218 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:17:50.023187 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:17:50.031885 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:17:50.031866 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:18:13.062990 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.062915 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:18:13.063478 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.063367 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerName="maas-api" Jul 10 21:18:13.063478 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.063381 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerName="maas-api" Jul 10 21:18:13.063478 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.063442 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="476b7d40-bcc1-41ba-80e2-a0d29a94b12f" containerName="maas-api" Jul 10 21:18:13.066568 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.066523 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.068713 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.068687 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-42c194-serving-cert\"" Jul 10 21:18:13.081458 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.081433 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:18:13.234482 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.234447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.234634 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.234496 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.234634 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.234580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.335419 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.335343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.335419 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.335384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.335590 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.335424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.335590 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:18:13.335508 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-serving-cert: secret "maas-api-e2e-inf-a-42c194-serving-cert" not found Jul 10 21:18:13.335590 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:18:13.335568 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls podName:d896fbe4-88be-40d3-903d-6915c0447512 nodeName:}" failed. No retries permitted until 2026-07-10 21:18:13.835551087 +0000 UTC m=+2424.442307646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls") pod "maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" (UID: "d896fbe4-88be-40d3-903d-6915c0447512") : secret "maas-api-e2e-inf-a-42c194-serving-cert" not found Jul 10 21:18:13.335975 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.335956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.345922 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.345893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.840825 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.840787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.843564 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.843532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") pod \"maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:13.976362 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:13.976332 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:14.123633 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:14.123607 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:18:14.125707 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:18:14.125680 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd896fbe4_88be_40d3_903d_6915c0447512.slice/crio-f50b4e39d03242856816f7a80f94784e8207d37dbb5f9b92d4a09da65cee49fd WatchSource:0}: Error finding container f50b4e39d03242856816f7a80f94784e8207d37dbb5f9b92d4a09da65cee49fd: Status 404 returned error can't find the container with id f50b4e39d03242856816f7a80f94784e8207d37dbb5f9b92d4a09da65cee49fd Jul 10 21:18:14.973644 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:14.973612 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" event={"ID":"d896fbe4-88be-40d3-903d-6915c0447512","Type":"ContainerStarted","Data":"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938"} Jul 10 21:18:14.973811 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:14.973670 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" event={"ID":"d896fbe4-88be-40d3-903d-6915c0447512","Type":"ContainerStarted","Data":"f50b4e39d03242856816f7a80f94784e8207d37dbb5f9b92d4a09da65cee49fd"} Jul 10 21:18:14.973811 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:14.973696 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:14.997202 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:14.997145 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" podStartSLOduration=1.501145648 podStartE2EDuration="1.997125475s" totalImagesPullingTime="495.979827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:18:14.99347821 +0000 UTC m=+2425.600234833" watchObservedRunningTime="2026-07-10 21:18:14.997125475 +0000 UTC m=+2425.603882056" Jul 10 21:18:20.984141 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:20.984110 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:18:21.264202 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.264131 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:18:21.268464 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.268438 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.270481 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.270458 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5f5b2e-serving-cert\"" Jul 10 21:18:21.277701 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.277676 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:18:21.311297 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.311210 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.311436 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.311384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpx8t\" (UniqueName: \"kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.311492 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.311475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.412059 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.412026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.412219 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.412091 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.412219 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.412161 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpx8t\" (UniqueName: \"kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.412219 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:18:21.412187 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-serving-cert: secret "maas-api-e2e-inf-b-5f5b2e-serving-cert" not found Jul 10 21:18:21.412394 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:18:21.412298 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls podName:5d7caa1c-2063-420e-8657-81f3e64e1263 nodeName:}" failed. No retries permitted until 2026-07-10 21:18:21.912261808 +0000 UTC m=+2432.519018367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls") pod "maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" (UID: "5d7caa1c-2063-420e-8657-81f3e64e1263") : secret "maas-api-e2e-inf-b-5f5b2e-serving-cert" not found Jul 10 21:18:21.412743 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.412724 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.422002 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.421981 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpx8t\" (UniqueName: \"kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.917416 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.917383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:21.920020 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:21.920000 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:22.180492 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:22.180396 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:22.320700 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:22.320668 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:18:22.323385 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:18:22.323349 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7caa1c_2063_420e_8657_81f3e64e1263.slice/crio-9e1e2f6ea06a88b3535debbf79a3ed7d65dffab570fa6ae62d3ab94907adb0e0 WatchSource:0}: Error finding container 9e1e2f6ea06a88b3535debbf79a3ed7d65dffab570fa6ae62d3ab94907adb0e0: Status 404 returned error can't find the container with id 9e1e2f6ea06a88b3535debbf79a3ed7d65dffab570fa6ae62d3ab94907adb0e0 Jul 10 21:18:23.018475 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:23.018437 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" event={"ID":"5d7caa1c-2063-420e-8657-81f3e64e1263","Type":"ContainerStarted","Data":"9e1e2f6ea06a88b3535debbf79a3ed7d65dffab570fa6ae62d3ab94907adb0e0"} Jul 10 21:18:24.025925 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:24.025885 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" event={"ID":"5d7caa1c-2063-420e-8657-81f3e64e1263","Type":"ContainerStarted","Data":"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8"} Jul 10 21:18:24.026342 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:24.026041 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:18:24.042291 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:24.042225 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" podStartSLOduration=2.245230098 podStartE2EDuration="3.042205809s" totalImagesPullingTime="796.975711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:18:24.041134808 +0000 UTC m=+2434.647891390" watchObservedRunningTime="2026-07-10 21:18:24.042205809 +0000 UTC m=+2434.648962391" Jul 10 21:18:30.036118 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:18:30.036088 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:19:45.753449 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:45.753411 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:19:45.753979 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:45.753660 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" podUID="d896fbe4-88be-40d3-903d-6915c0447512" containerName="maas-api" containerID="cri-o://525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938" gracePeriod=30 Jul 10 21:19:45.979563 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:45.979511 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" podUID="d896fbe4-88be-40d3-903d-6915c0447512" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.52:8443/health\": dial tcp 10.134.0.52:8443: connect: connection refused" Jul 10 21:19:46.220417 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.220392 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:19:46.368439 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.368351 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") pod \"d896fbe4-88be-40d3-903d-6915c0447512\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " Jul 10 21:19:46.368596 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.368463 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca\") pod \"d896fbe4-88be-40d3-903d-6915c0447512\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " Jul 10 21:19:46.368596 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.368521 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn\") pod \"d896fbe4-88be-40d3-903d-6915c0447512\" (UID: \"d896fbe4-88be-40d3-903d-6915c0447512\") " Jul 10 21:19:46.368852 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.368821 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d896fbe4-88be-40d3-903d-6915c0447512" (UID: "d896fbe4-88be-40d3-903d-6915c0447512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:19:46.370768 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.370741 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d896fbe4-88be-40d3-903d-6915c0447512" (UID: "d896fbe4-88be-40d3-903d-6915c0447512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:19:46.370768 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.370761 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn" (OuterVolumeSpecName: "kube-api-access-jpkqn") pod "d896fbe4-88be-40d3-903d-6915c0447512" (UID: "d896fbe4-88be-40d3-903d-6915c0447512"). InnerVolumeSpecName "kube-api-access-jpkqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:19:46.415700 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.415666 2585 generic.go:356] "Generic (PLEG): container finished" podID="d896fbe4-88be-40d3-903d-6915c0447512" containerID="525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938" exitCode=0 Jul 10 21:19:46.415825 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.415715 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" Jul 10 21:19:46.415825 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.415746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" event={"ID":"d896fbe4-88be-40d3-903d-6915c0447512","Type":"ContainerDied","Data":"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938"} Jul 10 21:19:46.415825 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.415800 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g" event={"ID":"d896fbe4-88be-40d3-903d-6915c0447512","Type":"ContainerDied","Data":"f50b4e39d03242856816f7a80f94784e8207d37dbb5f9b92d4a09da65cee49fd"} Jul 10 21:19:46.415934 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.415822 2585 scope.go:117] "RemoveContainer" containerID="525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938" Jul 10 21:19:46.425467 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.425449 2585 scope.go:117] "RemoveContainer" containerID="525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938" Jul 10 21:19:46.425698 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:19:46.425679 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938\": container with ID starting with 525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938 not found: ID does not exist" containerID="525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938" Jul 10 21:19:46.425795 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.425708 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938"} err="failed to get container status \"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938\": rpc error: code = NotFound desc = could not find container \"525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938\": container with ID starting with 525088771a25c11e6796dc6c4d6d0218926554f784f09020c01f84549d263938 not found: ID does not exist" Jul 10 21:19:46.442315 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.442288 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:19:46.446597 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.446578 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42c194-69fb5bb66c-5bq5g"] Jul 10 21:19:46.470086 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.470063 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d896fbe4-88be-40d3-903d-6915c0447512-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:19:46.470086 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.470084 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/d896fbe4-88be-40d3-903d-6915c0447512-kube-api-access-jpkqn\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:19:46.470201 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:46.470095 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d896fbe4-88be-40d3-903d-6915c0447512-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:19:47.985598 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:19:47.985563 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d896fbe4-88be-40d3-903d-6915c0447512" path="/var/lib/kubelet/pods/d896fbe4-88be-40d3-903d-6915c0447512/volumes" Jul 10 21:20:12.044641 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.044111 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:20:12.044641 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.044417 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" podUID="5d7caa1c-2063-420e-8657-81f3e64e1263" containerName="maas-api" containerID="cri-o://858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8" gracePeriod=30 Jul 10 21:20:12.293224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.293200 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:20:12.397739 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.397640 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") pod \"5d7caa1c-2063-420e-8657-81f3e64e1263\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " Jul 10 21:20:12.397739 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.397714 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca\") pod \"5d7caa1c-2063-420e-8657-81f3e64e1263\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " Jul 10 21:20:12.397973 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.397744 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpx8t\" (UniqueName: \"kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t\") pod \"5d7caa1c-2063-420e-8657-81f3e64e1263\" (UID: \"5d7caa1c-2063-420e-8657-81f3e64e1263\") " Jul 10 21:20:12.398131 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.398105 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d7caa1c-2063-420e-8657-81f3e64e1263" (UID: "5d7caa1c-2063-420e-8657-81f3e64e1263"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:20:12.399969 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.399937 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d7caa1c-2063-420e-8657-81f3e64e1263" (UID: "5d7caa1c-2063-420e-8657-81f3e64e1263"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:20:12.400128 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.400112 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t" (OuterVolumeSpecName: "kube-api-access-fpx8t") pod "5d7caa1c-2063-420e-8657-81f3e64e1263" (UID: "5d7caa1c-2063-420e-8657-81f3e64e1263"). InnerVolumeSpecName "kube-api-access-fpx8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:20:12.499827 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.499768 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7caa1c-2063-420e-8657-81f3e64e1263-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:20:12.499827 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.499811 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7caa1c-2063-420e-8657-81f3e64e1263-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:20:12.499827 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.499829 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpx8t\" (UniqueName: \"kubernetes.io/projected/5d7caa1c-2063-420e-8657-81f3e64e1263-kube-api-access-fpx8t\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:20:12.542997 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.542964 2585 generic.go:356] "Generic (PLEG): container finished" podID="5d7caa1c-2063-420e-8657-81f3e64e1263" containerID="858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8" exitCode=0 Jul 10 21:20:12.543178 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.543018 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" event={"ID":"5d7caa1c-2063-420e-8657-81f3e64e1263","Type":"ContainerDied","Data":"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8"} Jul 10 21:20:12.543178 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.543032 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" Jul 10 21:20:12.543178 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.543080 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk" event={"ID":"5d7caa1c-2063-420e-8657-81f3e64e1263","Type":"ContainerDied","Data":"9e1e2f6ea06a88b3535debbf79a3ed7d65dffab570fa6ae62d3ab94907adb0e0"} Jul 10 21:20:12.543178 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.543101 2585 scope.go:117] "RemoveContainer" containerID="858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8" Jul 10 21:20:12.552565 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.552549 2585 scope.go:117] "RemoveContainer" containerID="858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8" Jul 10 21:20:12.552845 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:20:12.552824 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8\": container with ID starting with 858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8 not found: ID does not exist" containerID="858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8" Jul 10 21:20:12.552928 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.552851 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8"} err="failed to get container status \"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8\": rpc error: code = NotFound desc = could not find container \"858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8\": container with ID starting with 858c3fdecc1544d65d9b7f9011d3d9d20c8ea8cdfd2c95e528fcd9ab618e37c8 not found: ID does not exist" Jul 10 21:20:12.563404 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.563381 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:20:12.569550 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:12.569528 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f5b2e-79dc4bd898-k75gk"] Jul 10 21:20:13.985312 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:13.985273 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7caa1c-2063-420e-8657-81f3e64e1263" path="/var/lib/kubelet/pods/5d7caa1c-2063-420e-8657-81f3e64e1263/volumes" Jul 10 21:20:31.769986 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.769948 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:20:31.770562 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770544 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d896fbe4-88be-40d3-903d-6915c0447512" containerName="maas-api" Jul 10 21:20:31.770605 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770566 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d896fbe4-88be-40d3-903d-6915c0447512" containerName="maas-api" Jul 10 21:20:31.770640 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770605 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7caa1c-2063-420e-8657-81f3e64e1263" containerName="maas-api" Jul 10 21:20:31.770640 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770614 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7caa1c-2063-420e-8657-81f3e64e1263" containerName="maas-api" Jul 10 21:20:31.770722 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770709 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d896fbe4-88be-40d3-903d-6915c0447512" containerName="maas-api" Jul 10 21:20:31.770772 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.770733 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7caa1c-2063-420e-8657-81f3e64e1263" containerName="maas-api" Jul 10 21:20:31.773818 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.773798 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.776071 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.776050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ed1347-serving-cert\"" Jul 10 21:20:31.787442 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.787417 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:20:31.878588 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.878550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.878588 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.878591 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.878832 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.878619 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwk7m\" (UniqueName: \"kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.979206 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.979178 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.979406 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.979212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.979406 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.979269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwk7m\" (UniqueName: \"kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.979823 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.979797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.981817 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.981787 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:31.999002 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:31.998974 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwk7m\" (UniqueName: \"kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m\") pod \"maas-api-e2e-api-a-ed1347-567d684589-vnmmm\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:32.084871 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:32.084768 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:32.231737 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:32.231711 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:20:32.234008 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:20:32.233977 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e875a8e_c31d_4fe3_8dff_b81af73490ac.slice/crio-4af3eef390bf4527de1c127709c9e72cb20e178b4d7f0138c9204d3dc141294a WatchSource:0}: Error finding container 4af3eef390bf4527de1c127709c9e72cb20e178b4d7f0138c9204d3dc141294a: Status 404 returned error can't find the container with id 4af3eef390bf4527de1c127709c9e72cb20e178b4d7f0138c9204d3dc141294a Jul 10 21:20:32.235402 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:32.235380 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:20:32.640803 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:32.640765 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" event={"ID":"1e875a8e-c31d-4fe3-8dff-b81af73490ac","Type":"ContainerStarted","Data":"4af3eef390bf4527de1c127709c9e72cb20e178b4d7f0138c9204d3dc141294a"} Jul 10 21:20:33.647587 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:33.647547 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" event={"ID":"1e875a8e-c31d-4fe3-8dff-b81af73490ac","Type":"ContainerStarted","Data":"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687"} Jul 10 21:20:33.647992 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:33.647664 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:33.666104 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:33.666047 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" podStartSLOduration=2.16562171 podStartE2EDuration="2.666026318s" totalImagesPullingTime="500.404608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:20:33.663921983 +0000 UTC m=+2564.270678586" watchObservedRunningTime="2026-07-10 21:20:33.666026318 +0000 UTC m=+2564.272782899" Jul 10 21:20:39.658555 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.658520 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:20:39.748807 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.748769 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:20:39.752956 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.752935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.755430 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.755409 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c4ebda-serving-cert\"" Jul 10 21:20:39.767058 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.766701 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:20:39.854374 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.854344 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbh8\" (UniqueName: \"kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.854523 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.854393 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.854523 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.854410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.955607 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.955567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbh8\" (UniqueName: \"kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.955764 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.955653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.955764 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.955679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.955898 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:20:39.955792 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-serving-cert: secret "maas-api-e2e-api-b-c4ebda-serving-cert" not found Jul 10 21:20:39.955898 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:20:39.955868 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls podName:9f9477f0-cb6d-41bc-9c93-9cda27417c17 nodeName:}" failed. No retries permitted until 2026-07-10 21:20:40.455847597 +0000 UTC m=+2571.062604157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls") pod "maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" (UID: "9f9477f0-cb6d-41bc-9c93-9cda27417c17") : secret "maas-api-e2e-api-b-c4ebda-serving-cert" not found Jul 10 21:20:39.956425 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.956373 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:39.966622 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:39.966598 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbh8\" (UniqueName: \"kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:40.461030 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:40.460998 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:40.463643 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:40.463621 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") pod \"maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:40.671539 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:40.671508 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:40.813564 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:40.813536 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:20:40.815655 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:20:40.815630 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9477f0_cb6d_41bc_9c93_9cda27417c17.slice/crio-202e97508bac1185f111eea1385fb9c6760901e2b67a5c8abf33f082d756dd3f WatchSource:0}: Error finding container 202e97508bac1185f111eea1385fb9c6760901e2b67a5c8abf33f082d756dd3f: Status 404 returned error can't find the container with id 202e97508bac1185f111eea1385fb9c6760901e2b67a5c8abf33f082d756dd3f Jul 10 21:20:41.689159 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:41.689123 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" event={"ID":"9f9477f0-cb6d-41bc-9c93-9cda27417c17","Type":"ContainerStarted","Data":"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4"} Jul 10 21:20:41.689568 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:41.689178 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" event={"ID":"9f9477f0-cb6d-41bc-9c93-9cda27417c17","Type":"ContainerStarted","Data":"202e97508bac1185f111eea1385fb9c6760901e2b67a5c8abf33f082d756dd3f"} Jul 10 21:20:41.689568 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:41.689267 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:20:41.707222 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:41.707174 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" podStartSLOduration=2.278188936 podStartE2EDuration="2.707159212s" totalImagesPullingTime="428.970276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:20:41.705108482 +0000 UTC m=+2572.311865087" watchObservedRunningTime="2026-07-10 21:20:41.707159212 +0000 UTC m=+2572.313915815" Jul 10 21:20:47.700437 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:20:47.700398 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:21:03.014882 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.014838 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:21:03.016006 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.015103 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" podUID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" containerName="maas-api" containerID="cri-o://e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687" gracePeriod=30 Jul 10 21:21:03.270984 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.270923 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:21:03.359293 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.359261 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca\") pod \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " Jul 10 21:21:03.359448 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.359311 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls\") pod \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " Jul 10 21:21:03.359448 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.359362 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwk7m\" (UniqueName: \"kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m\") pod \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\" (UID: \"1e875a8e-c31d-4fe3-8dff-b81af73490ac\") " Jul 10 21:21:03.359704 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.359663 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e875a8e-c31d-4fe3-8dff-b81af73490ac" (UID: "1e875a8e-c31d-4fe3-8dff-b81af73490ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:21:03.361572 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.361544 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e875a8e-c31d-4fe3-8dff-b81af73490ac" (UID: "1e875a8e-c31d-4fe3-8dff-b81af73490ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:21:03.361692 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.361674 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m" (OuterVolumeSpecName: "kube-api-access-zwk7m") pod "1e875a8e-c31d-4fe3-8dff-b81af73490ac" (UID: "1e875a8e-c31d-4fe3-8dff-b81af73490ac"). InnerVolumeSpecName "kube-api-access-zwk7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:21:03.460224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.460188 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e875a8e-c31d-4fe3-8dff-b81af73490ac-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:03.460224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.460220 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e875a8e-c31d-4fe3-8dff-b81af73490ac-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:03.460451 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.460255 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwk7m\" (UniqueName: \"kubernetes.io/projected/1e875a8e-c31d-4fe3-8dff-b81af73490ac-kube-api-access-zwk7m\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:03.799041 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.799001 2585 generic.go:356] "Generic (PLEG): container finished" podID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" containerID="e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687" exitCode=0 Jul 10 21:21:03.799224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.799054 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" event={"ID":"1e875a8e-c31d-4fe3-8dff-b81af73490ac","Type":"ContainerDied","Data":"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687"} Jul 10 21:21:03.799224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.799067 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" Jul 10 21:21:03.799224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.799098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm" event={"ID":"1e875a8e-c31d-4fe3-8dff-b81af73490ac","Type":"ContainerDied","Data":"4af3eef390bf4527de1c127709c9e72cb20e178b4d7f0138c9204d3dc141294a"} Jul 10 21:21:03.799224 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.799120 2585 scope.go:117] "RemoveContainer" containerID="e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687" Jul 10 21:21:03.815909 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.815888 2585 scope.go:117] "RemoveContainer" containerID="e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687" Jul 10 21:21:03.816194 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:21:03.816175 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687\": container with ID starting with e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687 not found: ID does not exist" containerID="e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687" Jul 10 21:21:03.816278 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.816203 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687"} err="failed to get container status \"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687\": rpc error: code = NotFound desc = could not find container \"e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687\": container with ID starting with e9b38cb10d4191e3167c7777fedd4ba33a0b64ae05ed8b5cc034199c38963687 not found: ID does not exist" Jul 10 21:21:03.823367 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.823342 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:21:03.827403 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.827377 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed1347-567d684589-vnmmm"] Jul 10 21:21:03.985266 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:03.985216 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" path="/var/lib/kubelet/pods/1e875a8e-c31d-4fe3-8dff-b81af73490ac/volumes" Jul 10 21:21:28.903010 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:28.902915 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:21:28.903670 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:28.903183 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" podUID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" containerName="maas-api" containerID="cri-o://8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4" gracePeriod=30 Jul 10 21:21:29.157350 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.157295 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:21:29.290758 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.290728 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca\") pod \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " Jul 10 21:21:29.290920 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.290807 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") pod \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " Jul 10 21:21:29.290920 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.290847 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rbh8\" (UniqueName: \"kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8\") pod \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\" (UID: \"9f9477f0-cb6d-41bc-9c93-9cda27417c17\") " Jul 10 21:21:29.291080 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.291056 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f9477f0-cb6d-41bc-9c93-9cda27417c17" (UID: "9f9477f0-cb6d-41bc-9c93-9cda27417c17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:21:29.292797 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.292769 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f9477f0-cb6d-41bc-9c93-9cda27417c17" (UID: "9f9477f0-cb6d-41bc-9c93-9cda27417c17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:21:29.292896 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.292873 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8" (OuterVolumeSpecName: "kube-api-access-9rbh8") pod "9f9477f0-cb6d-41bc-9c93-9cda27417c17" (UID: "9f9477f0-cb6d-41bc-9c93-9cda27417c17"). InnerVolumeSpecName "kube-api-access-9rbh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:21:29.392483 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.392453 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9477f0-cb6d-41bc-9c93-9cda27417c17-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:29.392483 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.392480 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rbh8\" (UniqueName: \"kubernetes.io/projected/9f9477f0-cb6d-41bc-9c93-9cda27417c17-kube-api-access-9rbh8\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:29.392483 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.392492 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9477f0-cb6d-41bc-9c93-9cda27417c17-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:21:29.931354 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.931312 2585 generic.go:356] "Generic (PLEG): container finished" podID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" containerID="8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4" exitCode=0 Jul 10 21:21:29.931755 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.931365 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" Jul 10 21:21:29.931755 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.931403 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" event={"ID":"9f9477f0-cb6d-41bc-9c93-9cda27417c17","Type":"ContainerDied","Data":"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4"} Jul 10 21:21:29.931755 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.931465 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl" event={"ID":"9f9477f0-cb6d-41bc-9c93-9cda27417c17","Type":"ContainerDied","Data":"202e97508bac1185f111eea1385fb9c6760901e2b67a5c8abf33f082d756dd3f"} Jul 10 21:21:29.931755 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.931493 2585 scope.go:117] "RemoveContainer" containerID="8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4" Jul 10 21:21:29.941674 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.941656 2585 scope.go:117] "RemoveContainer" containerID="8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4" Jul 10 21:21:29.941959 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:21:29.941938 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4\": container with ID starting with 8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4 not found: ID does not exist" containerID="8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4" Jul 10 21:21:29.942011 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.941968 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4"} err="failed to get container status \"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4\": rpc error: code = NotFound desc = could not find container \"8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4\": container with ID starting with 8b237b71bd7b1b045e49b5fb0b349b44fe2a3efe4dbb3ba586c6064e5037f0a4 not found: ID does not exist" Jul 10 21:21:29.956037 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.956007 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:21:29.959511 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.959489 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4ebda-d99c8dd8f-q97tl"] Jul 10 21:21:29.986116 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:29.986093 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" path="/var/lib/kubelet/pods/9f9477f0-cb6d-41bc-9c93-9cda27417c17/volumes" Jul 10 21:21:48.187493 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187455 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:21:48.187922 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187836 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" containerName="maas-api" Jul 10 21:21:48.187922 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187847 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" containerName="maas-api" Jul 10 21:21:48.187922 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187856 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" containerName="maas-api" Jul 10 21:21:48.187922 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187862 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" containerName="maas-api" Jul 10 21:21:48.188069 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187934 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e875a8e-c31d-4fe3-8dff-b81af73490ac" containerName="maas-api" Jul 10 21:21:48.188069 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.187943 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f9477f0-cb6d-41bc-9c93-9cda27417c17" containerName="maas-api" Jul 10 21:21:48.192335 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.192306 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.196088 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.196063 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-920ed7-serving-cert\"" Jul 10 21:21:48.201384 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.201279 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:21:48.256347 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.256315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.256553 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.256357 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.256553 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.256446 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4c2w\" (UniqueName: \"kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.357698 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.357660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4c2w\" (UniqueName: \"kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.357900 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.357717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.357900 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.357750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.358532 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.358510 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.360336 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.360318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.370927 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.370902 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4c2w\" (UniqueName: \"kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w\") pod \"maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.506897 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.506853 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:48.659699 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:48.659672 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:21:48.665006 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:21:48.664965 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398168a8_b7f0_49fe_ae83_aac1985e29f5.slice/crio-2542862a7f59f9998c8f1c54eb9793374373bb5fcd25b07be43236b67b586909 WatchSource:0}: Error finding container 2542862a7f59f9998c8f1c54eb9793374373bb5fcd25b07be43236b67b586909: Status 404 returned error can't find the container with id 2542862a7f59f9998c8f1c54eb9793374373bb5fcd25b07be43236b67b586909 Jul 10 21:21:49.025885 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:49.025854 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" event={"ID":"398168a8-b7f0-49fe-ae83-aac1985e29f5","Type":"ContainerStarted","Data":"2542862a7f59f9998c8f1c54eb9793374373bb5fcd25b07be43236b67b586909"} Jul 10 21:21:50.031368 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:50.031325 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" event={"ID":"398168a8-b7f0-49fe-ae83-aac1985e29f5","Type":"ContainerStarted","Data":"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce"} Jul 10 21:21:50.031828 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:50.031425 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:50.051644 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:50.051590 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" podStartSLOduration=1.721125641 podStartE2EDuration="2.051577902s" totalImagesPullingTime="330.452261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:50.049951513 +0000 UTC m=+2640.656708095" watchObservedRunningTime="2026-07-10 21:21:50.051577902 +0000 UTC m=+2640.658334483" Jul 10 21:21:56.043040 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.043007 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:21:56.177597 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.177559 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:21:56.181518 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.181492 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.183496 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.183470 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6e676e-serving-cert\"" Jul 10 21:21:56.193398 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.193369 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:21:56.230079 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.230034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.230308 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.230134 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twp57\" (UniqueName: \"kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.230308 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.230168 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.331618 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.331521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twp57\" (UniqueName: \"kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.331618 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.331566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.331809 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.331628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.331809 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:21:56.331759 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-serving-cert: secret "maas-api-e2e-shared-b-6e676e-serving-cert" not found Jul 10 21:21:56.331879 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:21:56.331860 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls podName:cadc2216-0765-444a-a507-87a84b549420 nodeName:}" failed. No retries permitted until 2026-07-10 21:21:56.831839045 +0000 UTC m=+2647.438595605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls") pod "maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" (UID: "cadc2216-0765-444a-a507-87a84b549420") : secret "maas-api-e2e-shared-b-6e676e-serving-cert" not found Jul 10 21:21:56.332387 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.332365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.340031 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.340005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twp57\" (UniqueName: \"kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.835723 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.835692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:56.838106 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:56.838080 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") pod \"maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:57.092736 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:57.092648 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:57.236050 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:57.236018 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:21:57.237426 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:21:57.237390 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadc2216_0765_444a_a507_87a84b549420.slice/crio-feaa43863efd7c1797974b9024778c6586144dff8819cbd88497ad5512d11426 WatchSource:0}: Error finding container feaa43863efd7c1797974b9024778c6586144dff8819cbd88497ad5512d11426: Status 404 returned error can't find the container with id feaa43863efd7c1797974b9024778c6586144dff8819cbd88497ad5512d11426 Jul 10 21:21:58.073948 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:58.073914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" event={"ID":"cadc2216-0765-444a-a507-87a84b549420","Type":"ContainerStarted","Data":"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66"} Jul 10 21:21:58.074138 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:58.073967 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" event={"ID":"cadc2216-0765-444a-a507-87a84b549420","Type":"ContainerStarted","Data":"feaa43863efd7c1797974b9024778c6586144dff8819cbd88497ad5512d11426"} Jul 10 21:21:58.074138 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:58.074041 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:21:58.093378 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:21:58.093316 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" podStartSLOduration=1.722444745 podStartE2EDuration="2.093295988s" totalImagesPullingTime="370.851243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:21:58.091622731 +0000 UTC m=+2648.698379334" watchObservedRunningTime="2026-07-10 21:21:58.093295988 +0000 UTC m=+2648.700052575" Jul 10 21:22:04.084117 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:22:04.084083 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:22:50.057442 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:22:50.057416 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:22:50.067182 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:22:50.067159 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:27:34.120752 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.120659 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:34.124402 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.124378 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:34.126473 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.126449 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-psb7f\"" Jul 10 21:27:34.127846 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.127824 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:34.192355 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.192316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc9l\" (UniqueName: \"kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l\") pod \"test-curl-7584-140217588129840\" (UID: \"e6d28668-f789-464d-8bd5-7ee55b5ee4e2\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:34.293880 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.293830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc9l\" (UniqueName: \"kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l\") pod \"test-curl-7584-140217588129840\" (UID: \"e6d28668-f789-464d-8bd5-7ee55b5ee4e2\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:34.301634 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.301595 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc9l\" (UniqueName: \"kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l\") pod \"test-curl-7584-140217588129840\" (UID: \"e6d28668-f789-464d-8bd5-7ee55b5ee4e2\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:34.435501 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.435413 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:34.572889 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.572863 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:34.574945 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:27:34.574915 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6d28668_f789_464d_8bd5_7ee55b5ee4e2.slice/crio-65014c6c586d5f8b484a3b220d8deda1a4bf4c1a11df751cd93827ef2f9608a8 WatchSource:0}: Error finding container 65014c6c586d5f8b484a3b220d8deda1a4bf4c1a11df751cd93827ef2f9608a8: Status 404 returned error can't find the container with id 65014c6c586d5f8b484a3b220d8deda1a4bf4c1a11df751cd93827ef2f9608a8 Jul 10 21:27:34.576273 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.576257 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:27:34.718400 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:34.718365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7584-140217588129840" event={"ID":"e6d28668-f789-464d-8bd5-7ee55b5ee4e2","Type":"ContainerStarted","Data":"65014c6c586d5f8b484a3b220d8deda1a4bf4c1a11df751cd93827ef2f9608a8"} Jul 10 21:27:35.724168 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:35.724133 2585 generic.go:356] "Generic (PLEG): container finished" podID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" containerID="772192e39044600eb5ea46600501adffcbdffb2befa0493d56c9f64c2c88f39a" exitCode=0 Jul 10 21:27:35.724566 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:35.724222 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7584-140217588129840" event={"ID":"e6d28668-f789-464d-8bd5-7ee55b5ee4e2","Type":"ContainerDied","Data":"772192e39044600eb5ea46600501adffcbdffb2befa0493d56c9f64c2c88f39a"} Jul 10 21:27:36.881195 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:36.881169 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:36.913745 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:36.913720 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7584-140217588129840_e6d28668-f789-464d-8bd5-7ee55b5ee4e2/test-curl-7584-140217588129840/0.log" Jul 10 21:27:36.938431 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:36.938399 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:36.946393 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:36.946370 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:37.020071 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.019978 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncc9l\" (UniqueName: \"kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l\") pod \"e6d28668-f789-464d-8bd5-7ee55b5ee4e2\" (UID: \"e6d28668-f789-464d-8bd5-7ee55b5ee4e2\") " Jul 10 21:27:37.022211 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.022181 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l" (OuterVolumeSpecName: "kube-api-access-ncc9l") pod "e6d28668-f789-464d-8bd5-7ee55b5ee4e2" (UID: "e6d28668-f789-464d-8bd5-7ee55b5ee4e2"). InnerVolumeSpecName "kube-api-access-ncc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:27:37.080089 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.080057 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:37.080476 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.080462 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" containerName="test-curl-7584-140217588129840" Jul 10 21:27:37.080526 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.080477 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" containerName="test-curl-7584-140217588129840" Jul 10 21:27:37.080562 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.080537 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" containerName="test-curl-7584-140217588129840" Jul 10 21:27:37.083581 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.083563 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.089937 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.089912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:37.121451 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.121425 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncc9l\" (UniqueName: \"kubernetes.io/projected/e6d28668-f789-464d-8bd5-7ee55b5ee4e2-kube-api-access-ncc9l\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:27:37.222587 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.222562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whswm\" (UniqueName: \"kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm\") pod \"test-curl-7584-140217588129840\" (UID: \"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.323934 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.323850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whswm\" (UniqueName: \"kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm\") pod \"test-curl-7584-140217588129840\" (UID: \"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.334045 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.334023 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whswm\" (UniqueName: \"kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm\") pod \"test-curl-7584-140217588129840\" (UID: \"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9\") " pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.393116 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.393092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.528736 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.528706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:37.530126 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:27:37.530097 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09805fa6_27a6_4d93_bc91_f1dbaa8bcae9.slice/crio-129ee9c71dd9bfe8b1d22821ee95ec45dcdc60d195afb490cb06708a263547cd WatchSource:0}: Error finding container 129ee9c71dd9bfe8b1d22821ee95ec45dcdc60d195afb490cb06708a263547cd: Status 404 returned error can't find the container with id 129ee9c71dd9bfe8b1d22821ee95ec45dcdc60d195afb490cb06708a263547cd Jul 10 21:27:37.734875 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.734841 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7584-140217588129840" event={"ID":"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9","Type":"ContainerStarted","Data":"129ee9c71dd9bfe8b1d22821ee95ec45dcdc60d195afb490cb06708a263547cd"} Jul 10 21:27:37.736067 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.736044 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65014c6c586d5f8b484a3b220d8deda1a4bf4c1a11df751cd93827ef2f9608a8" Jul 10 21:27:37.736067 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.736051 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:37.747141 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.747114 2585 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7584-140217588129840" oldPodUID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" podUID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" Jul 10 21:27:37.986653 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:37.986580 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d28668-f789-464d-8bd5-7ee55b5ee4e2" path="/var/lib/kubelet/pods/e6d28668-f789-464d-8bd5-7ee55b5ee4e2/volumes" Jul 10 21:27:38.741870 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:38.741834 2585 generic.go:356] "Generic (PLEG): container finished" podID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" containerID="52f4abf5e5483c00265d4e215f6f07be239595cc39b67a3c55b96b059ff687e5" exitCode=0 Jul 10 21:27:38.742027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:38.741879 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7584-140217588129840" event={"ID":"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9","Type":"ContainerDied","Data":"52f4abf5e5483c00265d4e215f6f07be239595cc39b67a3c55b96b059ff687e5"} Jul 10 21:27:39.895727 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:39.895705 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:39.918169 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:39.918135 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7584-140217588129840_09805fa6-27a6-4d93-bc91-f1dbaa8bcae9/test-curl-7584-140217588129840/0.log" Jul 10 21:27:39.945544 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:39.945510 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:39.949368 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:39.949346 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7584-140217588129840"] Jul 10 21:27:40.047882 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:40.047806 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whswm\" (UniqueName: \"kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm\") pod \"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9\" (UID: \"09805fa6-27a6-4d93-bc91-f1dbaa8bcae9\") " Jul 10 21:27:40.049937 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:40.049912 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm" (OuterVolumeSpecName: "kube-api-access-whswm") pod "09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" (UID: "09805fa6-27a6-4d93-bc91-f1dbaa8bcae9"). InnerVolumeSpecName "kube-api-access-whswm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:27:40.149203 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:40.149173 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whswm\" (UniqueName: \"kubernetes.io/projected/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9-kube-api-access-whswm\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:27:40.752248 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:40.752199 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7584-140217588129840" Jul 10 21:27:40.752446 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:40.752202 2585 scope.go:117] "RemoveContainer" containerID="52f4abf5e5483c00265d4e215f6f07be239595cc39b67a3c55b96b059ff687e5" Jul 10 21:27:41.985136 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:41.985102 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" path="/var/lib/kubelet/pods/09805fa6-27a6-4d93-bc91-f1dbaa8bcae9/volumes" Jul 10 21:27:50.086356 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:50.086328 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:27:50.096048 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:50.096025 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxw5l_8c85bfd1-981f-4a23-9eb8-70e5f3c57a1c/ovn-acl-logging/0.log" Jul 10 21:27:56.751106 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:56.751062 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:27:56.751635 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:56.751423 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" podUID="398168a8-b7f0-49fe-ae83-aac1985e29f5" containerName="maas-api" containerID="cri-o://8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce" gracePeriod=30 Jul 10 21:27:57.012247 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.012170 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:27:57.097348 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.096675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca\") pod \"398168a8-b7f0-49fe-ae83-aac1985e29f5\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " Jul 10 21:27:57.097348 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.096732 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4c2w\" (UniqueName: \"kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w\") pod \"398168a8-b7f0-49fe-ae83-aac1985e29f5\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " Jul 10 21:27:57.097348 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.096796 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls\") pod \"398168a8-b7f0-49fe-ae83-aac1985e29f5\" (UID: \"398168a8-b7f0-49fe-ae83-aac1985e29f5\") " Jul 10 21:27:57.097348 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.097135 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "398168a8-b7f0-49fe-ae83-aac1985e29f5" (UID: "398168a8-b7f0-49fe-ae83-aac1985e29f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:27:57.097348 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.097296 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/398168a8-b7f0-49fe-ae83-aac1985e29f5-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:27:57.099300 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.099255 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w" (OuterVolumeSpecName: "kube-api-access-d4c2w") pod "398168a8-b7f0-49fe-ae83-aac1985e29f5" (UID: "398168a8-b7f0-49fe-ae83-aac1985e29f5"). InnerVolumeSpecName "kube-api-access-d4c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:27:57.099558 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.099526 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "398168a8-b7f0-49fe-ae83-aac1985e29f5" (UID: "398168a8-b7f0-49fe-ae83-aac1985e29f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:27:57.197930 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.197884 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/398168a8-b7f0-49fe-ae83-aac1985e29f5-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:27:57.197930 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.197928 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4c2w\" (UniqueName: \"kubernetes.io/projected/398168a8-b7f0-49fe-ae83-aac1985e29f5-kube-api-access-d4c2w\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:27:57.843133 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.843099 2585 generic.go:356] "Generic (PLEG): container finished" podID="398168a8-b7f0-49fe-ae83-aac1985e29f5" containerID="8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce" exitCode=0 Jul 10 21:27:57.843543 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.843155 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" Jul 10 21:27:57.843543 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.843183 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" event={"ID":"398168a8-b7f0-49fe-ae83-aac1985e29f5","Type":"ContainerDied","Data":"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce"} Jul 10 21:27:57.843543 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.843287 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc" event={"ID":"398168a8-b7f0-49fe-ae83-aac1985e29f5","Type":"ContainerDied","Data":"2542862a7f59f9998c8f1c54eb9793374373bb5fcd25b07be43236b67b586909"} Jul 10 21:27:57.843543 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.843322 2585 scope.go:117] "RemoveContainer" containerID="8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce" Jul 10 21:27:57.852935 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.852915 2585 scope.go:117] "RemoveContainer" containerID="8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce" Jul 10 21:27:57.853200 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:27:57.853178 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce\": container with ID starting with 8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce not found: ID does not exist" containerID="8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce" Jul 10 21:27:57.853296 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.853213 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce"} err="failed to get container status \"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce\": rpc error: code = NotFound desc = could not find container \"8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce\": container with ID starting with 8b8f3ff190b7f6f401761303acdae0ef2f283568bf9523d4cd44de99e05e72ce not found: ID does not exist" Jul 10 21:27:57.863836 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.863811 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:27:57.869818 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.869799 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-920ed7-f48d96cc9-vgqtc"] Jul 10 21:27:57.985799 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:27:57.985768 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398168a8-b7f0-49fe-ae83-aac1985e29f5" path="/var/lib/kubelet/pods/398168a8-b7f0-49fe-ae83-aac1985e29f5/volumes" Jul 10 21:28:23.082834 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.082795 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:28:23.083439 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.083066 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" podUID="cadc2216-0765-444a-a507-87a84b549420" containerName="maas-api" containerID="cri-o://cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66" gracePeriod=30 Jul 10 21:28:23.330747 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.330722 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:28:23.417876 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.417801 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") pod \"cadc2216-0765-444a-a507-87a84b549420\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " Jul 10 21:28:23.417876 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.417837 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twp57\" (UniqueName: \"kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57\") pod \"cadc2216-0765-444a-a507-87a84b549420\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " Jul 10 21:28:23.417876 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.417870 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca\") pod \"cadc2216-0765-444a-a507-87a84b549420\" (UID: \"cadc2216-0765-444a-a507-87a84b549420\") " Jul 10 21:28:23.418293 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.418262 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cadc2216-0765-444a-a507-87a84b549420" (UID: "cadc2216-0765-444a-a507-87a84b549420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:28:23.420024 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.419998 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cadc2216-0765-444a-a507-87a84b549420" (UID: "cadc2216-0765-444a-a507-87a84b549420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:28:23.420130 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.420032 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57" (OuterVolumeSpecName: "kube-api-access-twp57") pod "cadc2216-0765-444a-a507-87a84b549420" (UID: "cadc2216-0765-444a-a507-87a84b549420"). InnerVolumeSpecName "kube-api-access-twp57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:28:23.518715 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.518689 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cadc2216-0765-444a-a507-87a84b549420-maas-api-tls\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:28:23.518715 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.518713 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twp57\" (UniqueName: \"kubernetes.io/projected/cadc2216-0765-444a-a507-87a84b549420-kube-api-access-twp57\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:28:23.518870 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.518725 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadc2216-0765-444a-a507-87a84b549420-openshift-service-ca\") on node \"ip-10-0-142-194.ec2.internal\" DevicePath \"\"" Jul 10 21:28:23.971475 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.971437 2585 generic.go:356] "Generic (PLEG): container finished" podID="cadc2216-0765-444a-a507-87a84b549420" containerID="cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66" exitCode=0 Jul 10 21:28:23.971671 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.971481 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" Jul 10 21:28:23.971671 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.971522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" event={"ID":"cadc2216-0765-444a-a507-87a84b549420","Type":"ContainerDied","Data":"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66"} Jul 10 21:28:23.971671 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.971579 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs" event={"ID":"cadc2216-0765-444a-a507-87a84b549420","Type":"ContainerDied","Data":"feaa43863efd7c1797974b9024778c6586144dff8819cbd88497ad5512d11426"} Jul 10 21:28:23.971671 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.971611 2585 scope.go:117] "RemoveContainer" containerID="cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66" Jul 10 21:28:23.981820 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.981804 2585 scope.go:117] "RemoveContainer" containerID="cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66" Jul 10 21:28:23.982142 ip-10-0-142-194 kubenswrapper[2585]: E0710 21:28:23.982123 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66\": container with ID starting with cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66 not found: ID does not exist" containerID="cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66" Jul 10 21:28:23.982338 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:23.982157 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66"} err="failed to get container status \"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66\": rpc error: code = NotFound desc = could not find container \"cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66\": container with ID starting with cea52c94dcf33124f79b729aea4be761391e8484461db12e98417dbaaa9c6c66 not found: ID does not exist" Jul 10 21:28:24.000383 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:24.000354 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:28:24.002067 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:24.002043 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6e676e-69cf6865f4-j6ghs"] Jul 10 21:28:25.985811 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:25.985776 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cadc2216-0765-444a-a507-87a84b549420" path="/var/lib/kubelet/pods/cadc2216-0765-444a-a507-87a84b549420/volumes" Jul 10 21:28:43.809519 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:43.809439 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gtd22_e2bceca8-b4f5-4902-83f4-07b9b062f51b/manager/0.log" Jul 10 21:28:43.933138 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:43.933106 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-k84td_b8ccc476-bce0-4524-a89b-821a1e924a54/manager/0.log" Jul 10 21:28:44.057138 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:44.057097 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9b4b8b8c-hlvpm_cf65154a-eafc-45c9-8937-0461da6e60ef/manager/0.log" Jul 10 21:28:44.168991 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:44.168900 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mgpmz_e23a37d2-5644-4d39-bb48-eb7ddb047705/server/0.log" Jul 10 21:28:44.292100 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:44.292071 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xhdhj_cc8a48f1-5618-4d1c-b298-00a7ec08f796/manager/0.log" Jul 10 21:28:44.659587 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:44.659562 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6558bffdbc-vsh7x_64818ea5-07a6-46fc-a9f9-a92f07a3d717/copy-manifests/0.log" Jul 10 21:28:44.677472 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:44.677446 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6558bffdbc-vsh7x_64818ea5-07a6-46fc-a9f9-a92f07a3d717/manager/0.log" Jul 10 21:28:45.060287 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:45.060207 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5846b788db-zqdvz_e8714a9e-1dc5-46bc-9ed3-a6c808ce7426/maas-api/0.log" Jul 10 21:28:46.757375 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:46.757340 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hwtmh_3e00795f-8e91-48a7-bce8-389c05d63636/postgres/0.log" Jul 10 21:28:48.160863 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:48.160832 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-r6rf8_b2e1a486-d121-40fb-a324-47c4d84fa158/manager/0.log" Jul 10 21:28:48.273908 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:48.273879 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-njzfb_f1719f05-e730-4a7c-a11b-2656a92be333/manager/0.log" Jul 10 21:28:48.384527 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:48.384473 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xt7rl_57d40615-bb8e-45f8-8ad6-20a837d190a5/kuadrant-console-plugin/0.log" Jul 10 21:28:48.510583 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:48.510553 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kdf75_ddf41688-34ed-40e0-8ceb-50f9c32cb95e/registry-server/0.log" Jul 10 21:28:48.639641 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:48.639599 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xsbts_88af342a-9612-447a-bf8e-25ca984a716d/manager/0.log" Jul 10 21:28:49.575358 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:49.575330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86d9dbc65b-nm2qq_90aa0555-e386-4867-ae3b-00bec97eb705/kube-auth-proxy/0.log" Jul 10 21:28:49.825482 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:49.825392 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dpg96_0665b549-3067-4ef6-95fc-30d32cf32915/payload-pre-processing/0.log" Jul 10 21:28:49.952091 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:49.952059 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-px95v_deddbc3e-1c04-4cd5-b218-8aa8a445b09a/payload-processing/0.log" Jul 10 21:28:50.423360 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:50.423326 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qj5w7_45b1b509-cede-47f9-b55a-9ec8ceef1c0b/main/0.log" Jul 10 21:28:50.540335 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:50.540243 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7txpf_f848b66c-b3ed-48d7-800c-798a1a947837/main/0.log" Jul 10 21:28:50.763118 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:50.763076 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdt25v_573d8a7d-5df9-4d22-9cf3-8a548183d045/main/0.log" Jul 10 21:28:50.874479 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:50.874407 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4f9rh_8f1e55fc-8362-4419-8c58-b033df969e06/main/0.log" Jul 10 21:28:58.020161 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:58.020132 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdr8l_1842cdd3-d259-4106-9d65-618540a99480/global-pull-secret-syncer/0.log" Jul 10 21:28:58.117529 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:58.117496 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rmqm6_a989078a-a850-4db3-9042-d195a3e461e7/konnectivity-agent/0.log" Jul 10 21:28:58.212463 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:28:58.212433 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-194.ec2.internal_197bfbeb65b7318ec9e668e91130f11b/haproxy/0.log" Jul 10 21:29:03.326382 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:03.326356 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-r6rf8_b2e1a486-d121-40fb-a324-47c4d84fa158/manager/0.log" Jul 10 21:29:03.349064 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:03.349038 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-njzfb_f1719f05-e730-4a7c-a11b-2656a92be333/manager/0.log" Jul 10 21:29:03.375049 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:03.375028 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xt7rl_57d40615-bb8e-45f8-8ad6-20a837d190a5/kuadrant-console-plugin/0.log" Jul 10 21:29:03.439355 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:03.439321 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kdf75_ddf41688-34ed-40e0-8ceb-50f9c32cb95e/registry-server/0.log" Jul 10 21:29:03.606342 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:03.606253 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xsbts_88af342a-9612-447a-bf8e-25ca984a716d/manager/0.log" Jul 10 21:29:05.823569 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:05.823540 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tddc_d269469c-ec62-4c52-907d-bfc8256b81f6/node-exporter/0.log" Jul 10 21:29:05.838508 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:05.838480 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tddc_d269469c-ec62-4c52-907d-bfc8256b81f6/kube-rbac-proxy/0.log" Jul 10 21:29:05.854928 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:05.854908 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tddc_d269469c-ec62-4c52-907d-bfc8256b81f6/init-textfile/0.log" Jul 10 21:29:06.336150 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:06.336116 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fdb797dbd-p86t2_bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8/telemeter-client/0.log" Jul 10 21:29:06.352764 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:06.352738 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fdb797dbd-p86t2_bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8/reload/0.log" Jul 10 21:29:06.368805 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:06.368784 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fdb797dbd-p86t2_bcabdaab-ba60-4aa7-a1f9-f49f54ddd0c8/kube-rbac-proxy/0.log" Jul 10 21:29:08.188650 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.188622 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p"] Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.188966 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" containerName="test-curl-7584-140217588129840" Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.188977 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" containerName="test-curl-7584-140217588129840" Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.188987 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398168a8-b7f0-49fe-ae83-aac1985e29f5" containerName="maas-api" Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.188992 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="398168a8-b7f0-49fe-ae83-aac1985e29f5" containerName="maas-api" Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.189005 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadc2216-0765-444a-a507-87a84b549420" containerName="maas-api" Jul 10 21:29:08.189027 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.189010 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadc2216-0765-444a-a507-87a84b549420" containerName="maas-api" Jul 10 21:29:08.189254 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.189071 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="09805fa6-27a6-4d93-bc91-f1dbaa8bcae9" containerName="test-curl-7584-140217588129840" Jul 10 21:29:08.189254 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.189081 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cadc2216-0765-444a-a507-87a84b549420" containerName="maas-api" Jul 10 21:29:08.189254 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.189087 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="398168a8-b7f0-49fe-ae83-aac1985e29f5" containerName="maas-api" Jul 10 21:29:08.192087 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.192071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.194044 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.194022 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhd5q\"/\"kube-root-ca.crt\"" Jul 10 21:29:08.194409 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.194380 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bhd5q\"/\"default-dockercfg-blt7c\"" Jul 10 21:29:08.194409 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.194390 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhd5q\"/\"openshift-service-ca.crt\"" Jul 10 21:29:08.200370 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.200349 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p"] Jul 10 21:29:08.213258 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.213222 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-lib-modules\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.213343 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.213263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq42g\" (UniqueName: \"kubernetes.io/projected/6ca42996-5e0c-48a4-80a0-0f736f5e5793-kube-api-access-kq42g\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.213343 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.213297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-proc\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.213343 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.213327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-sys\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.213444 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.213349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-podres\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314415 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-lib-modules\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314415 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq42g\" (UniqueName: \"kubernetes.io/projected/6ca42996-5e0c-48a4-80a0-0f736f5e5793-kube-api-access-kq42g\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-proc\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-sys\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-podres\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-proc\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-sys\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-lib-modules\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.314647 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.314627 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca42996-5e0c-48a4-80a0-0f736f5e5793-podres\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.322570 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.322550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq42g\" (UniqueName: \"kubernetes.io/projected/6ca42996-5e0c-48a4-80a0-0f736f5e5793-kube-api-access-kq42g\") pod \"perf-node-gather-daemonset-nft4p\" (UID: \"6ca42996-5e0c-48a4-80a0-0f736f5e5793\") " pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.502872 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.502785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:08.631031 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:08.631006 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p"] Jul 10 21:29:08.632886 ip-10-0-142-194 kubenswrapper[2585]: W0710 21:29:08.632859 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ca42996_5e0c_48a4_80a0_0f736f5e5793.slice/crio-c81890720ec2bcd7137cdd924a60d7775216d38f8a7fe2bde801e62a8feffabe WatchSource:0}: Error finding container c81890720ec2bcd7137cdd924a60d7775216d38f8a7fe2bde801e62a8feffabe: Status 404 returned error can't find the container with id c81890720ec2bcd7137cdd924a60d7775216d38f8a7fe2bde801e62a8feffabe Jul 10 21:29:09.184510 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:09.184473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" event={"ID":"6ca42996-5e0c-48a4-80a0-0f736f5e5793","Type":"ContainerStarted","Data":"e4ae2a3581c9346c625ef6d016a6dea734d5fb15c6515037636fb09b222f0dff"} Jul 10 21:29:09.184747 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:09.184529 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" event={"ID":"6ca42996-5e0c-48a4-80a0-0f736f5e5793","Type":"ContainerStarted","Data":"c81890720ec2bcd7137cdd924a60d7775216d38f8a7fe2bde801e62a8feffabe"} Jul 10 21:29:09.184747 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:09.184560 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:09.198908 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:09.198862 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" podStartSLOduration=1.198844399 podStartE2EDuration="1.198844399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:29:09.198374175 +0000 UTC m=+3079.805130783" watchObservedRunningTime="2026-07-10 21:29:09.198844399 +0000 UTC m=+3079.805600982" Jul 10 21:29:10.444389 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:10.444358 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n66mb_ce5621c9-1df4-4e9d-87a1-335f497c90bc/dns/0.log" Jul 10 21:29:10.469386 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:10.469359 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n66mb_ce5621c9-1df4-4e9d-87a1-335f497c90bc/kube-rbac-proxy/0.log" Jul 10 21:29:10.553130 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:10.553103 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zzgw_cfa3b044-cb6f-41f8-b5d8-96fe2948004c/dns-node-resolver/0.log" Jul 10 21:29:11.098943 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:11.098899 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84fb596665-kl4sk_d8eb31fc-c5b5-48ce-8458-3cd8ed838cb5/registry/0.log" Jul 10 21:29:11.140572 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:11.140548 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6xrm_cb239305-035b-4c95-bda0-aabdb6f65431/node-ca/0.log" Jul 10 21:29:12.314955 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:12.314930 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86d9dbc65b-nm2qq_90aa0555-e386-4867-ae3b-00bec97eb705/kube-auth-proxy/0.log" Jul 10 21:29:12.447569 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:12.447533 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dpg96_0665b549-3067-4ef6-95fc-30d32cf32915/payload-pre-processing/0.log" Jul 10 21:29:12.484037 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:12.484011 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-px95v_deddbc3e-1c04-4cd5-b218-8aa8a445b09a/payload-processing/0.log" Jul 10 21:29:13.168110 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:13.168077 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2kbcj_f9d1ff0b-f1ed-410b-9655-fe196396be09/serve-healthcheck-canary/0.log" Jul 10 21:29:13.668528 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:13.668502 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2r288_6fee6004-aff7-41a0-822b-bb6bae06d810/kube-rbac-proxy/0.log" Jul 10 21:29:13.684839 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:13.684812 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2r288_6fee6004-aff7-41a0-822b-bb6bae06d810/exporter/0.log" Jul 10 21:29:13.702154 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:13.702131 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2r288_6fee6004-aff7-41a0-822b-bb6bae06d810/extractor/0.log" Jul 10 21:29:15.198564 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:15.198538 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bhd5q/perf-node-gather-daemonset-nft4p" Jul 10 21:29:15.991710 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:15.991677 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gtd22_e2bceca8-b4f5-4902-83f4-07b9b062f51b/manager/0.log" Jul 10 21:29:16.230724 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.230688 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-k84td_b8ccc476-bce0-4524-a89b-821a1e924a54/manager/0.log" Jul 10 21:29:16.395345 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.395257 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d9b4b8b8c-hlvpm_cf65154a-eafc-45c9-8937-0461da6e60ef/manager/0.log" Jul 10 21:29:16.413867 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.413846 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mgpmz_e23a37d2-5644-4d39-bb48-eb7ddb047705/server/0.log" Jul 10 21:29:16.516253 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.516200 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xhdhj_cc8a48f1-5618-4d1c-b298-00a7ec08f796/manager/0.log" Jul 10 21:29:16.746272 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.746218 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6558bffdbc-vsh7x_64818ea5-07a6-46fc-a9f9-a92f07a3d717/manager/0.log" Jul 10 21:29:16.764193 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:16.764169 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6558bffdbc-vsh7x_64818ea5-07a6-46fc-a9f9-a92f07a3d717/copy-manifests/0.log" Jul 10 21:29:18.181728 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:18.181698 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x77px_02d86451-732e-4ae7-ad02-08a619fc0de4/openshift-lws-operator/0.log" Jul 10 21:29:18.679988 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:18.679955 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86d9dbc65b-nm2qq_90aa0555-e386-4867-ae3b-00bec97eb705/kube-auth-proxy/0.log" Jul 10 21:29:18.787248 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:18.787197 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dpg96_0665b549-3067-4ef6-95fc-30d32cf32915/payload-pre-processing/0.log" Jul 10 21:29:18.815545 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:18.815519 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-px95v_deddbc3e-1c04-4cd5-b218-8aa8a445b09a/payload-processing/0.log" Jul 10 21:29:20.255432 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:20.255402 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86d9dbc65b-nm2qq_90aa0555-e386-4867-ae3b-00bec97eb705/kube-auth-proxy/0.log" Jul 10 21:29:20.373899 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:20.373871 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dpg96_0665b549-3067-4ef6-95fc-30d32cf32915/payload-pre-processing/0.log" Jul 10 21:29:20.405452 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:20.405422 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-px95v_deddbc3e-1c04-4cd5-b218-8aa8a445b09a/payload-processing/0.log" Jul 10 21:29:24.563080 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.563052 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/kube-multus-additional-cni-plugins/0.log" Jul 10 21:29:24.579617 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.579591 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/egress-router-binary-copy/0.log" Jul 10 21:29:24.594793 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.594771 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/cni-plugins/0.log" Jul 10 21:29:24.611649 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.611629 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/bond-cni-plugin/0.log" Jul 10 21:29:24.626834 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.626811 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/routeoverride-cni/0.log" Jul 10 21:29:24.641670 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.641645 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/whereabouts-cni-bincopy/0.log" Jul 10 21:29:24.657256 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.657211 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dl78j_a57761bd-4dbe-4ba8-8bd6-1bd424c2c4b0/whereabouts-cni/0.log" Jul 10 21:29:24.862526 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.862452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dgdgm_56ff23da-1f36-4fc2-b33e-230ba5252990/kube-multus/0.log" Jul 10 21:29:24.965372 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.965344 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vnrsl_f525f08f-e462-48e9-8530-85773208a545/network-metrics-daemon/0.log" Jul 10 21:29:24.977400 ip-10-0-142-194 kubenswrapper[2585]: I0710 21:29:24.977376 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vnrsl_f525f08f-e462-48e9-8530-85773208a545/kube-rbac-proxy/0.log"