Jul 16 12:05:04.381295 ip-10-0-139-22 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 12:05:04.381306 ip-10-0-139-22 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 12:05:04.381313 ip-10-0-139-22 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 12:05:04.381625 ip-10-0-139-22 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 12:05:13.724188 ip-10-0-139-22 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 12:05:13.724202 ip-10-0-139-22 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bb650eb056464a2e992413dddb78317c -- Jul 16 12:06:42.171272 ip-10-0-139-22 systemd[1]: Starting Kubernetes Kubelet... Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.631960 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636414 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636428 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636431 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636435 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636438 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636441 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636444 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:42.806579 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636447 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:42.705424 ip-10-0-139-22 systemd[1]: Started Kubernetes Kubelet. Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636450 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636453 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636456 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636459 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636461 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636464 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636467 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636470 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636472 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636475 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636478 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636481 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636483 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636486 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636488 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636497 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636499 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636502 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:42.955176 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636505 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636508 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636511 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636513 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636516 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636518 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636521 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636523 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636526 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636528 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636531 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636533 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636535 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636538 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636540 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636544 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636547 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636549 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636554 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:42.956215 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636557 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636560 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636563 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636567 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636571 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636574 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636577 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636580 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636582 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636585 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636587 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636591 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636594 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636596 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636599 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636602 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636605 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636608 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636611 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:42.956884 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636616 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636619 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636622 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636625 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636627 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636630 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636632 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636635 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636638 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636640 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636643 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636646 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636649 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636651 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636654 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636656 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636659 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636661 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636664 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636666 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:42.957521 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636668 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636672 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636674 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636677 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636680 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636682 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636686 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636689 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636692 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636694 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636697 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636699 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636702 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636705 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.636708 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637417 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637424 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637427 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637430 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:42.958361 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637432 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637435 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637438 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637441 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637444 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637446 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637449 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637452 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637455 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637457 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637460 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637463 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637466 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637468 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637471 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637473 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637476 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637481 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637486 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:42.958972 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637489 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637493 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637497 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637500 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637504 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637507 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637511 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637514 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637517 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637519 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637522 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637525 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637527 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637529 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637532 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637536 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637538 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637541 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637543 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637546 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:42.959822 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637548 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637551 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637553 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637555 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637558 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637561 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637563 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637566 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637568 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637571 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637574 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637576 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637579 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637582 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637586 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637588 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637591 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637594 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637596 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:42.960630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637599 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637602 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637604 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637607 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637609 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637612 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637614 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637616 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637619 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637626 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637628 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637631 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637634 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637636 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637639 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637641 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637644 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637646 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637649 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637652 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:42.961237 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637654 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637657 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637659 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637663 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637667 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637670 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637673 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637678 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637681 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637683 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637686 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637688 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637691 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637695 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637698 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637700 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.637703 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638064 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638079 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:42.961874 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638082 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638085 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638088 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638090 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638093 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638098 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638101 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638103 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638106 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638109 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638112 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638114 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638117 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638121 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638125 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638128 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638131 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638133 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638136 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:42.962523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638139 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638142 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638146 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638149 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638151 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638154 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638156 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638159 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638162 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638164 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638167 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638169 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638172 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638176 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638180 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638183 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638186 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638189 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638193 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638196 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:42.963082 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638198 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638200 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638204 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638207 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638209 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638212 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638214 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638217 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638219 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638221 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638224 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638227 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638229 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638232 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638235 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638238 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638240 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638242 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638245 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638247 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:42.963689 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638250 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638252 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638255 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638257 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638259 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638262 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638265 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638267 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638269 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638272 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638276 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638279 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638281 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638284 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638286 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638289 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638291 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638294 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638296 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:42.964269 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638300 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638303 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638306 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638308 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638311 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638313 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638316 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638319 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638322 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638324 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638327 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638329 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638332 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638334 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638337 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638339 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638342 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638344 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.638347 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638417 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 12:06:42.964849 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638431 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638437 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638442 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638449 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638453 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638457 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638461 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638464 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638467 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638470 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638474 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638477 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638479 2583 flags.go:64] FLAG: --cgroup-root="" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638482 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638485 2583 flags.go:64] FLAG: --client-ca-file="" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638488 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638491 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638495 2583 flags.go:64] FLAG: --cluster-domain="" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638498 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638506 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638509 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638513 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638517 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 12:06:42.965403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638520 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638523 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638526 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638529 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638531 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638534 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638537 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638540 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638544 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638547 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638550 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638553 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638557 2583 flags.go:64] FLAG: --enable-server="true" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638560 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638564 2583 flags.go:64] FLAG: --event-burst="100" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638567 2583 flags.go:64] FLAG: --event-qps="50" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638570 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638572 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638576 2583 flags.go:64] FLAG: --eviction-hard="" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638579 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638582 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638585 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638588 2583 flags.go:64] FLAG: --eviction-soft="" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638591 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638594 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 12:06:42.966054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638597 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638600 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638603 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638607 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638610 2583 flags.go:64] FLAG: --feature-gates="" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638613 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638616 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638620 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638623 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638626 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638629 2583 flags.go:64] FLAG: --help="false" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638632 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-22.ec2.internal" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638635 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638638 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638640 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638644 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638648 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638651 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638653 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638658 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638661 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638664 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638667 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638670 2583 flags.go:64] FLAG: --kube-reserved="" Jul 16 12:06:42.966714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638673 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638676 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638679 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638682 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638685 2583 flags.go:64] FLAG: --lock-file="" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638687 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638690 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638693 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638698 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638701 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638704 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638708 2583 flags.go:64] FLAG: --logging-format="text" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638710 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638714 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638717 2583 flags.go:64] FLAG: --manifest-url="" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638719 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638723 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638726 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638730 2583 flags.go:64] FLAG: --max-pods="110" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638733 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638736 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638739 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638742 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638746 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638748 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 12:06:42.967562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638751 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638758 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638762 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638765 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638768 2583 flags.go:64] FLAG: --pod-cidr="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638772 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638777 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638780 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638783 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638786 2583 flags.go:64] FLAG: --port="10250" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638789 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638792 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd670794ccfb67a5" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638795 2583 flags.go:64] FLAG: --qos-reserved="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638798 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638801 2583 flags.go:64] FLAG: --register-node="true" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638804 2583 flags.go:64] FLAG: --register-with-taints="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638807 2583 flags.go:64] FLAG: --registry-burst="10" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638810 2583 flags.go:64] FLAG: --registry-qps="5" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638814 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638817 2583 flags.go:64] FLAG: --reserved-memory="" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638821 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638824 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638827 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638830 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638833 2583 flags.go:64] FLAG: --runonce="false" Jul 16 12:06:42.970714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638836 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638839 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638841 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638844 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638848 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638851 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638854 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638856 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638859 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638862 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638867 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638870 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638873 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638876 2583 flags.go:64] FLAG: --system-cgroups="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638879 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638884 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638888 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638890 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638895 2583 flags.go:64] FLAG: --tls-min-version="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638897 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638900 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638903 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638906 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638909 2583 flags.go:64] FLAG: --v="2" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638919 2583 flags.go:64] FLAG: --version="false" Jul 16 12:06:42.971544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638924 2583 flags.go:64] FLAG: --vmodule="" Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638927 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.638931 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639052 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639056 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639059 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639062 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639065 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639067 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639070 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639073 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639075 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639078 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639081 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639083 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639086 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639088 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639093 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639095 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639098 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:43.110282 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639101 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639104 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639106 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639109 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639111 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639114 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639117 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639120 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639123 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639125 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639128 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639130 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639134 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639137 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639139 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639142 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639144 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639147 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639149 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639151 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:43.110899 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639154 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639157 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639159 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639162 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639164 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639167 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639169 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639171 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639174 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639178 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639180 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639183 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639185 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639188 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639192 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639195 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639197 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639200 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639202 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:43.111448 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639205 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639207 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639210 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639213 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639215 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639219 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639222 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639224 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639227 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639229 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639232 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639234 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639237 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639239 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639242 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639245 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639247 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639250 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639253 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639257 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:43.111983 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639260 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639263 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639268 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639271 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639274 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639277 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639279 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639282 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639285 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639287 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639290 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639293 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639295 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639299 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639302 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639305 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639308 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639312 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639314 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:43.112529 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639317 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639320 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639322 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.639325 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.639330 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.646304 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.646318 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.646338 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.646344 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646399 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646404 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646407 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646411 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646413 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646416 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:43.113075 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646420 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646423 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646426 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646428 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646431 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646434 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646436 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646439 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646443 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646446 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646448 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646451 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646456 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646459 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646462 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646465 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646467 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646470 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646472 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646475 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:43.113547 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646478 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646481 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646483 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646486 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646489 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646491 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646494 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646496 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646499 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646501 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646504 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646506 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646509 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646513 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646517 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646521 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646525 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646527 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646530 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:43.114131 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646532 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646535 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646538 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646540 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646543 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646547 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646549 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646552 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646555 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646558 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646560 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646562 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646565 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646568 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646570 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646573 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646575 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646578 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646581 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646583 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:43.114618 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646586 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646588 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646591 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646593 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646596 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646599 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646602 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646604 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646608 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646611 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646613 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646616 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646618 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646622 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646626 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646629 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646632 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646638 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646640 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:43.115198 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646643 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646645 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646648 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646650 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646653 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646655 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646658 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646661 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646663 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646666 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646669 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646673 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646677 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646680 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:43.115691 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646682 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.646688 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646803 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646808 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646811 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646814 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646817 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646823 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646826 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646829 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646831 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646834 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646837 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646839 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646842 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:06:43.116116 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646844 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646846 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646851 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646855 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646858 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646861 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646863 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646866 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646869 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646872 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646875 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646877 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646880 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646883 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646885 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646888 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646891 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646893 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646896 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646898 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:06:43.116519 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646901 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646903 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646906 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646908 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646912 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646915 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646917 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646919 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646922 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646925 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646927 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646929 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646932 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646934 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646938 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646941 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646944 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646946 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646948 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:06:43.117089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646951 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646954 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646956 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646959 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646961 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646964 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646966 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646968 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646971 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646974 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646977 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646979 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646981 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.646984 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647000 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647003 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647006 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647011 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647013 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647016 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:06:43.117630 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647019 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647021 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647024 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647026 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647029 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647031 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647034 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647038 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647040 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647043 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647045 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647048 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647050 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647053 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647056 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647058 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647061 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647065 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647067 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:06:43.118196 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647070 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647073 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647076 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647079 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647082 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647085 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647088 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:42.647090 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.647095 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.647196 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.649492 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.651747 2583 server.go:1019] "Starting client certificate rotation" Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.651790 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:06:43.118679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.651814 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.657120 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.659851 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.669903 2583 log.go:25] "Validated CRI v1 runtime API" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.672891 2583 log.go:25] "Validated CRI v1 image API" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.673730 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.678226 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 adad990c-50d7-4cc0-87ab-b5c01f98e236:/dev/nvme0n1p4 afc9f6be-0a0c-44bf-991a-71c371622650:/dev/nvme0n1p3] Jul 16 12:06:43.119135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.678243 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.684435 2583 manager.go:216] Machine: {Timestamp:2026-07-16 12:06:42.68309281 +0000 UTC m=+0.377199540 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099068 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e98c249226dc9047485ef4d3e69a2 SystemUUID:ec2e98c2-4922-6dc9-0474-85ef4d3e69a2 BootID:bb650eb0-5646-4a2e-9924-13dddb78317c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:b8:c8:a9:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:b8:c8:a9:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:da:25:6c:4a:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.684536 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.684612 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.685640 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.685662 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.686729 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.686740 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.686747 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.686762 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.688628 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.689983 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692135 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692247 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692274 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692284 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692294 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 12:06:43.119421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692302 2583 kubelet.go:398] "Adding apiserver pod source" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692310 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692676 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.692700 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.697050 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699726 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699755 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699913 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699925 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699930 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699936 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699944 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.699950 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700847 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700854 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700861 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700867 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700875 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700883 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700906 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.700912 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.704548 2583 server.go:1268] "Started kubelet" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.704714 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.704719 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-22.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.704765 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.704824 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.704823 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.704876 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.706208 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.707424 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.710213 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-22.ec2.internal.18c2c3b93a2cdde3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-22.ec2.internal,UID:ip-10-0-139-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-22.ec2.internal,},FirstTimestamp:2026-07-16 12:06:42.704522723 +0000 UTC m=+0.398629433,LastTimestamp:2026-07-16 12:06:42.704522723 +0000 UTC m=+0.398629433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-22.ec2.internal,}" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.712878 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.716597 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.716605 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.716673 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717264 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717270 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717278 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717372 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717380 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717580 2583 factory.go:153] Registering CRI-O factory Jul 16 12:06:43.120223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717591 2583 factory.go:223] Registration of the crio container factory successfully Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717640 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717648 2583 factory.go:55] Registering systemd factory Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717654 2583 factory.go:223] Registration of the systemd container factory successfully Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717670 2583 factory.go:103] Registering Raw factory Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.717678 2583 manager.go:1196] Started watching for new ooms in manager Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.718041 2583 manager.go:349] Starting recovery of all containers Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.719185 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.719317 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.727977 2583 manager.go:354] Recovery completed Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.733660 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735436 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735465 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735479 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735921 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735933 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735938 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.735948 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.736642 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.736663 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.736672 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.737242 2583 policy_none.go:49] "None policy: Start" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.737258 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.737267 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.738373 2583 policy_none.go:47] "Start" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.744354 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-22.ec2.internal.18c2c3b93c04c556 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-22.ec2.internal,UID:ip-10-0-139-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-22.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-22.ec2.internal,},FirstTimestamp:2026-07-16 12:06:42.73544943 +0000 UTC m=+0.429556143,LastTimestamp:2026-07-16 12:06:42.73544943 +0000 UTC m=+0.429556143,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-22.ec2.internal,}" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.748437 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-22.ec2.internal.18c2c3b93c051d51 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-22.ec2.internal,UID:ip-10-0-139-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-22.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-22.ec2.internal,},FirstTimestamp:2026-07-16 12:06:42.735471953 +0000 UTC m=+0.429578663,LastTimestamp:2026-07-16 12:06:42.735471953 +0000 UTC m=+0.429578663,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-22.ec2.internal,}" Jul 16 12:06:43.121306 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.751723 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rz89z" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.756863 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-22.ec2.internal.18c2c3b93c054984 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-22.ec2.internal,UID:ip-10-0-139-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-22.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-22.ec2.internal,},FirstTimestamp:2026-07-16 12:06:42.735483268 +0000 UTC m=+0.429589981,LastTimestamp:2026-07-16 12:06:42.735483268 +0000 UTC m=+0.429589981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-22.ec2.internal,}" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.761918 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rz89z" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.786374 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.787527 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.787561 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.787577 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.787608 2583 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.793682 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.816712 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.826512 2583 manager.go:337] "Starting Device Plugin manager" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.826546 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.826555 2583 server.go:87] "Starting device plugin registration server" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.826771 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.826783 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.826976 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.827072 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.827080 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.827746 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.827790 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.888459 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal"] Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.888555 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.889577 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.889603 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.889612 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.891039 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.891180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.891210 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892510 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892530 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892541 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892576 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892595 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.892605 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.893854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.893882 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.894558 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.122661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.894586 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.894595 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.907563 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-22.ec2.internal\" not found" node="ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.911788 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-22.ec2.internal\" not found" node="ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.918563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.918584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.918602 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ae3cb7d0f0636b302c0e293270dd787-config\") pod \"kube-apiserver-proxy-ip-10-0-139-22.ec2.internal\" (UID: \"7ae3cb7d0f0636b302c0e293270dd787\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.925807 2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-22.ec2.internal\" not found" node="ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.927877 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.928639 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.928665 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.928675 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.928696 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:42.944717 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.944731 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-22.ec2.internal\": node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:42.991741 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019708 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019725 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef1df8b73c3bb3a4c595daeb07adb161-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal\" (UID: \"ef1df8b73c3bb3a4c595daeb07adb161\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ae3cb7d0f0636b302c0e293270dd787-config\") pod \"kube-apiserver-proxy-ip-10-0-139-22.ec2.internal\" (UID: \"7ae3cb7d0f0636b302c0e293270dd787\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.019780 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ae3cb7d0f0636b302c0e293270dd787-config\") pod \"kube-apiserver-proxy-ip-10-0-139-22.ec2.internal\" (UID: \"7ae3cb7d0f0636b302c0e293270dd787\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.123715 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.092289 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.221326 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.192911 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.221326 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.211173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.221326 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.215548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:43.301322 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.293981 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.443260 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.394933 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.537070 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.496026 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.704387 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.598724 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.704387 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.652245 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 12:06:43.704387 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.652418 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:06:43.704387 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.698787 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.847957 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.710236 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:06:43.847957 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.768321 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 12:01:43 +0000 UTC" deadline="2028-05-01 06:58:46.725323664 +0000 UTC" Jul 16 12:06:43.847957 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.768350 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15714h52m2.956976608s" Jul 16 12:06:43.847957 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.799673 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.876742 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:43.900583 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.913671 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.914757 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.914789 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:06:43.961019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:43.914803 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:06:44.094077 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.000659 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-22.ec2.internal\" not found" Jul 16 12:06:44.094077 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.030266 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:06:44.192602 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.117077 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" Jul 16 12:06:44.192602 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.134467 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:06:44.192602 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.135658 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" Jul 16 12:06:44.192602 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.168005 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:06:44.306127 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:44.218058 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1df8b73c3bb3a4c595daeb07adb161.slice/crio-2a2f2846ca51c760799f42306281efc79342e6e1d831b96bc0f724302939ca60 WatchSource:0}: Error finding container 2a2f2846ca51c760799f42306281efc79342e6e1d831b96bc0f724302939ca60: Status 404 returned error can't find the container with id 2a2f2846ca51c760799f42306281efc79342e6e1d831b96bc0f724302939ca60 Jul 16 12:06:44.306127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.228105 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:06:44.695849 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.695746 2583 apiserver.go:52] "Watching apiserver" Jul 16 12:06:44.706545 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.706520 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 12:06:44.706900 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.706878 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-rd4b2","kube-system/global-pull-secret-syncer-bzpqj","openshift-cluster-node-tuning-operator/tuned-nnvj8","openshift-dns/node-resolver-hbpk4","openshift-image-registry/node-ca-kb2qd","openshift-multus/multus-additional-cni-plugins-4wzhr","openshift-multus/multus-xd8m9","openshift-multus/network-metrics-daemon-4zs6r","openshift-network-operator/iptables-alerter-z9r8b","kube-system/konnectivity-agent-dsmnv","kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal","openshift-network-diagnostics/network-check-target-gcltf"] Jul 16 12:06:44.709294 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.709273 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.710555 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.710532 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.710684 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.710599 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:44.711723 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.711701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.712952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.712931 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.714076 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.714059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.714425 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.714198 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 12:06:44.714425 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.714309 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.714624 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.714607 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.714927 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.714904 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hzn77\"" Jul 16 12:06:44.715496 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.715476 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.715591 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.715529 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 12:06:44.716698 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.716680 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.717731 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.717714 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 12:06:44.718129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.718111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.718815 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.718798 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.719321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.719301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.719419 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.719369 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:44.720631 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.720609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.721450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721431 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.721564 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721541 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.721663 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721646 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.721867 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721845 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-66vtx\"" Jul 16 12:06:44.721963 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721908 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:44.722049 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.721965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:44.722049 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.721987 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 12:06:44.723403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.723386 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.731329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.731416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9pz\" (UniqueName: \"kubernetes.io/projected/3aa69c1f-fde7-408e-9175-aff918f86270-kube-api-access-9x9pz\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.731416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-kube-api-access-z94rg\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.731416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-netd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovn-node-metrics-cert\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731497 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-cni-binary-copy\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.731584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhbq\" (UniqueName: \"kubernetes.io/projected/5c7e88a7-710b-422c-9bae-a64e036e1001-kube-api-access-8jhbq\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.731584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-conf\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.731584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-etc-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731603 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-env-overrides\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731628 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-system-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731651 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-system-cni-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-config\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-node-log\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.731780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-log-socket\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-modprobe-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-kubernetes\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-tmp\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d320534-8242-4791-8292-7fbfca21d331-tmp-dir\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.731978 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6l2l\" (UniqueName: \"kubernetes.io/projected/1d320534-8242-4791-8292-7fbfca21d331-kube-api-access-c6l2l\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.732072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-run\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.732357 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.732357 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732101 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.732454 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-serviceca\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.732503 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-kubelet\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.732550 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-bin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.732601 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732574 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-kubelet\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.732651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732629 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-host\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.732705 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-bin\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.732984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-k8s-cni-cncf-io\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-lib-modules\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733063 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cnibin\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733094 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-ovn\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-os-release\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjtb\" (UniqueName: \"kubernetes.io/projected/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-kube-api-access-wnjtb\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnrh\" (UniqueName: \"kubernetes.io/projected/7628b79f-14bf-4fb4-83e2-b48f5208e310-kube-api-access-6nnrh\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-conf-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-systemd\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-slash\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-cnibin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysconfig\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733539 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-tuned\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-host\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-var-lib-kubelet\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-os-release\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-systemd-units\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-var-lib-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.733781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcd2\" (UniqueName: \"kubernetes.io/projected/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-kube-api-access-kwcd2\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7628b79f-14bf-4fb4-83e2-b48f5208e310-host-slash\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-socket-dir-parent\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-multus\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-sys\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-script-lib\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-daemon-config\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-multus-certs\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-etc-kubernetes\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqxl\" (UniqueName: \"kubernetes.io/projected/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-kube-api-access-rlqxl\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734656 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-netns\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-systemd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.735823 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d320534-8242-4791-8292-7fbfca21d331-hosts-file\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.736420 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734740 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7628b79f-14bf-4fb4-83e2-b48f5208e310-iptables-alerter-script\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.736420 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-netns\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.736420 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.734807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-hostroot\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.739145 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.739121 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.739145 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.739128 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 12:06:44.739299 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.739153 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.739969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.739936 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.740293 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740275 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 12:06:44.740376 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740293 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jkqj9\"" Jul 16 12:06:44.740376 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740306 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 12:06:44.740376 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740323 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dbrpz\"" Jul 16 12:06:44.740528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740391 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fc227\"" Jul 16 12:06:44.740528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740300 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.740528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740463 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.740528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.740506 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fggxn\"" Jul 16 12:06:44.741384 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741366 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 12:06:44.741457 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741366 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 12:06:44.741457 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741388 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 12:06:44.741569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741369 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.741690 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741674 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 12:06:44.741751 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741705 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 12:06:44.741806 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741755 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-29t4l\"" Jul 16 12:06:44.741806 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741790 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 12:06:44.741916 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741813 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 12:06:44.741916 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741865 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 12:06:44.741916 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741890 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bxc5v\"" Jul 16 12:06:44.742089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.741933 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 12:06:44.765394 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.765377 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vdv8q\"" Jul 16 12:06:44.771511 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.771493 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:06:44.788416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.788396 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:06:44.792785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.792746 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" event={"ID":"ef1df8b73c3bb3a4c595daeb07adb161","Type":"ContainerStarted","Data":"2a2f2846ca51c760799f42306281efc79342e6e1d831b96bc0f724302939ca60"} Jul 16 12:06:44.818817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.818796 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 12:06:44.836079 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysconfig\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-tuned\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-host\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.836196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75b86a55-a963-41ff-82a1-c528689346a3-konnectivity-ca\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.836196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-var-lib-kubelet\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-os-release\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836197 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysconfig\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-host\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836214 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-systemd-units\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836251 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-systemd-units\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836265 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-var-lib-kubelet\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-var-lib-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836287 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-os-release\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcd2\" (UniqueName: \"kubernetes.io/projected/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-kube-api-access-kwcd2\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7628b79f-14bf-4fb4-83e2-b48f5208e310-host-slash\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-var-lib-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-socket-dir-parent\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836403 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7628b79f-14bf-4fb4-83e2-b48f5208e310-host-slash\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-multus\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.836437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75b86a55-a963-41ff-82a1-c528689346a3-agent-certs\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-sys\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836476 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-multus\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836508 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-socket-dir-parent\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-script-lib\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836509 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-daemon-config\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836543 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-sys\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836542 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-multus-certs\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-etc-kubernetes\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-kubelet-config\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836643 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-etc-kubernetes\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836652 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-multus-certs\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836684 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqxl\" (UniqueName: \"kubernetes.io/projected/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-kube-api-access-rlqxl\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.837191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836703 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-netns\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-systemd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d320534-8242-4791-8292-7fbfca21d331-hosts-file\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836765 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7628b79f-14bf-4fb4-83e2-b48f5208e310-iptables-alerter-script\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-netns\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836799 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-netns\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-hostroot\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836832 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-systemd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-netns\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9pz\" (UniqueName: \"kubernetes.io/projected/3aa69c1f-fde7-408e-9175-aff918f86270-kube-api-access-9x9pz\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-kube-api-access-z94rg\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-netd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836977 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d320534-8242-4791-8292-7fbfca21d331-hosts-file\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.836987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovn-node-metrics-cert\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-hostroot\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-cni-binary-copy\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhbq\" (UniqueName: \"kubernetes.io/projected/5c7e88a7-710b-422c-9bae-a64e036e1001-kube-api-access-8jhbq\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837129 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-script-lib\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-conf\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-etc-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837205 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-daemon-config\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-env-overrides\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837206 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-system-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-system-cni-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837366 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-sysctl-conf\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-config\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837409 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-dbus\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837439 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7628b79f-14bf-4fb4-83e2-b48f5208e310-iptables-alerter-script\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.838952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-node-log\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837479 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-netd\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-log-socket\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.837505 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837544 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-modprobe-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.837589 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:45.337545896 +0000 UTC m=+3.031652612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-kubernetes\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837643 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-modprobe-d\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-tmp\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837684 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d320534-8242-4791-8292-7fbfca21d331-tmp-dir\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6l2l\" (UniqueName: \"kubernetes.io/projected/1d320534-8242-4791-8292-7fbfca21d331-kube-api-access-c6l2l\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-run\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837827 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837847 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.839847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-env-overrides\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837908 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-system-cni-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-serviceca\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837941 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c7e88a7-710b-422c-9bae-a64e036e1001-cni-binary-copy\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-kubelet\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.837504 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-etc-openvswitch\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838256 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d320534-8242-4791-8292-7fbfca21d331-tmp-dir\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-bin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-kubelet\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838393 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovnkube-config\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-serviceca\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-kubelet\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838486 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-cni-bin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-host\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838525 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-var-lib-kubelet\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-bin\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838543 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-node-log\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.840687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838591 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-log-socket\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-k8s-cni-cncf-io\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838612 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-host\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-lib-modules\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838631 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-host-run-k8s-cni-cncf-io\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-cni-bin\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-run\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838785 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-lib-modules\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838810 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-run-ovn-kubernetes\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838867 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-system-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.838956 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-kubernetes\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cnibin\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-ovn\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-os-release\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839219 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.841244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2b7r\" (UniqueName: \"kubernetes.io/projected/813c752f-d510-4ccc-8551-317d603d7cf1-kube-api-access-f2b7r\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjtb\" (UniqueName: \"kubernetes.io/projected/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-kube-api-access-wnjtb\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839309 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-os-release\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnrh\" (UniqueName: \"kubernetes.io/projected/7628b79f-14bf-4fb4-83e2-b48f5208e310-kube-api-access-6nnrh\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-cni-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cnibin\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-run-ovn\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-conf-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839364 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-multus-conf-dir\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-systemd\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839496 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-slash\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-cnibin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839548 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-systemd\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839580 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839600 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c7e88a7-710b-422c-9bae-a64e036e1001-cnibin\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.841725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.842262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839618 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-host-slash\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.842262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.839931 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.842262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.840742 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-etc-tuned\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.842262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.840775 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-tmp\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.842262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.841470 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-ovn-node-metrics-cert\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.848776 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.848761 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-549bk" Jul 16 12:06:44.849711 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.849695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcd2\" (UniqueName: \"kubernetes.io/projected/e4aa1f4d-445c-4753-a5dc-c7c5317fd337-kube-api-access-kwcd2\") pod \"ovnkube-node-rd4b2\" (UID: \"e4aa1f4d-445c-4753-a5dc-c7c5317fd337\") " pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:44.870669 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.870638 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhbq\" (UniqueName: \"kubernetes.io/projected/5c7e88a7-710b-422c-9bae-a64e036e1001-kube-api-access-8jhbq\") pod \"multus-xd8m9\" (UID: \"5c7e88a7-710b-422c-9bae-a64e036e1001\") " pod="openshift-multus/multus-xd8m9" Jul 16 12:06:44.871140 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.871124 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjtb\" (UniqueName: \"kubernetes.io/projected/b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973-kube-api-access-wnjtb\") pod \"multus-additional-cni-plugins-4wzhr\" (UID: \"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973\") " pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:44.871203 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.871127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnrh\" (UniqueName: \"kubernetes.io/projected/7628b79f-14bf-4fb4-83e2-b48f5208e310-kube-api-access-6nnrh\") pod \"iptables-alerter-z9r8b\" (UID: \"7628b79f-14bf-4fb4-83e2-b48f5208e310\") " pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:44.872178 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.872159 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-549bk" Jul 16 12:06:44.873509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.873492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6l2l\" (UniqueName: \"kubernetes.io/projected/1d320534-8242-4791-8292-7fbfca21d331-kube-api-access-c6l2l\") pod \"node-resolver-hbpk4\" (UID: \"1d320534-8242-4791-8292-7fbfca21d331\") " pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:44.873983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.873964 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94rg\" (UniqueName: \"kubernetes.io/projected/fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea-kube-api-access-z94rg\") pod \"tuned-nnvj8\" (UID: \"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea\") " pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:44.876007 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.875989 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9pz\" (UniqueName: \"kubernetes.io/projected/3aa69c1f-fde7-408e-9175-aff918f86270-kube-api-access-9x9pz\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:44.881451 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.881434 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqxl\" (UniqueName: \"kubernetes.io/projected/2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7-kube-api-access-rlqxl\") pod \"node-ca-kb2qd\" (UID: \"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7\") " pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:44.940553 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940530 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75b86a55-a963-41ff-82a1-c528689346a3-agent-certs\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940557 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-kubelet-config\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940645 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-kubelet-config\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.940662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-dbus\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.940692 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940708 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/470371ed-667c-4545-a836-883ce8e71ff5-dbus\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-registration-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-device-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.940745 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:45.440729244 +0000 UTC m=+3.134835946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940756 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2b7r\" (UniqueName: \"kubernetes.io/projected/813c752f-d510-4ccc-8551-317d603d7cf1-kube-api-access-f2b7r\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940849 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-sys-fs\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75b86a55-a963-41ff-82a1-c528689346a3-konnectivity-ca\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.940983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.940941 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.941579 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.941035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/813c752f-d510-4ccc-8551-317d603d7cf1-socket-dir\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:44.941579 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.941381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75b86a55-a963-41ff-82a1-c528689346a3-konnectivity-ca\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.942713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.942696 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75b86a55-a963-41ff-82a1-c528689346a3-agent-certs\") pod \"konnectivity-agent-dsmnv\" (UID: \"75b86a55-a963-41ff-82a1-c528689346a3\") " pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:44.964768 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.964710 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:06:44.964768 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.964733 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:06:44.964768 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.964749 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:44.964925 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:44.964805 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:45.464791481 +0000 UTC m=+3.158898179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:44.968984 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:44.968966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2b7r\" (UniqueName: \"kubernetes.io/projected/813c752f-d510-4ccc-8551-317d603d7cf1-kube-api-access-f2b7r\") pod \"aws-ebs-csi-driver-node-b6wsz\" (UID: \"813c752f-d510-4ccc-8551-317d603d7cf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:45.021004 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.020963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xd8m9" Jul 16 12:06:45.026124 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.026099 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" Jul 16 12:06:45.030423 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.030402 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7e88a7_710b_422c_9bae_a64e036e1001.slice/crio-f2d1d30117b4a151f61c8a5ff3704b0bd72f5dc86eb0ceaf9afa6d6095b80373 WatchSource:0}: Error finding container f2d1d30117b4a151f61c8a5ff3704b0bd72f5dc86eb0ceaf9afa6d6095b80373: Status 404 returned error can't find the container with id f2d1d30117b4a151f61c8a5ff3704b0bd72f5dc86eb0ceaf9afa6d6095b80373 Jul 16 12:06:45.033523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.033507 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbpk4" Jul 16 12:06:45.035495 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.035440 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceb94e3_aefe_49d8_8aec_cfef3ee7c1ea.slice/crio-043cd7104c7459a5d563a4326a223ae555ef64c958543ff1def7b377e3eeb98f WatchSource:0}: Error finding container 043cd7104c7459a5d563a4326a223ae555ef64c958543ff1def7b377e3eeb98f: Status 404 returned error can't find the container with id 043cd7104c7459a5d563a4326a223ae555ef64c958543ff1def7b377e3eeb98f Jul 16 12:06:45.038890 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.038868 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kb2qd" Jul 16 12:06:45.042218 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.042191 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d320534_8242_4791_8292_7fbfca21d331.slice/crio-541fc78a61d4822952f31e21573177a461f7d8d1f2c0544a7915f0bbae6a6bf8 WatchSource:0}: Error finding container 541fc78a61d4822952f31e21573177a461f7d8d1f2c0544a7915f0bbae6a6bf8: Status 404 returned error can't find the container with id 541fc78a61d4822952f31e21573177a461f7d8d1f2c0544a7915f0bbae6a6bf8 Jul 16 12:06:45.045643 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.045625 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" Jul 16 12:06:45.047496 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.047473 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6e0e6e_b9ae_454b_8c0f_ca76ccf970a7.slice/crio-2e90c03d165206e0967351704d1d282711660107e55ec198e72f2d8a833a6843 WatchSource:0}: Error finding container 2e90c03d165206e0967351704d1d282711660107e55ec198e72f2d8a833a6843: Status 404 returned error can't find the container with id 2e90c03d165206e0967351704d1d282711660107e55ec198e72f2d8a833a6843 Jul 16 12:06:45.052395 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.051864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9r8b" Jul 16 12:06:45.057897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.057873 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:06:45.058239 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.058214 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e7e3ad_4f02_4f06_bb10_d19e1d6e7973.slice/crio-aabce6b5d44d84bcc9c806c98f61a5ac9050275834ef5c267b6e83d1344777ed WatchSource:0}: Error finding container aabce6b5d44d84bcc9c806c98f61a5ac9050275834ef5c267b6e83d1344777ed: Status 404 returned error can't find the container with id aabce6b5d44d84bcc9c806c98f61a5ac9050275834ef5c267b6e83d1344777ed Jul 16 12:06:45.060148 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.060129 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7628b79f_14bf_4fb4_83e2_b48f5208e310.slice/crio-ea15ff152db8d3cc7791c5e4b106976e5153c4ad0b77e79409e2de2517911c78 WatchSource:0}: Error finding container ea15ff152db8d3cc7791c5e4b106976e5153c4ad0b77e79409e2de2517911c78: Status 404 returned error can't find the container with id ea15ff152db8d3cc7791c5e4b106976e5153c4ad0b77e79409e2de2517911c78 Jul 16 12:06:45.064155 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.064139 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:06:45.066139 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.066121 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aa1f4d_445c_4753_a5dc_c7c5317fd337.slice/crio-3b1c7dc6771f7384b860016de531e08f98af97013baa85c148aaaa5c4cd9fbf1 WatchSource:0}: Error finding container 3b1c7dc6771f7384b860016de531e08f98af97013baa85c148aaaa5c4cd9fbf1: Status 404 returned error can't find the container with id 3b1c7dc6771f7384b860016de531e08f98af97013baa85c148aaaa5c4cd9fbf1 Jul 16 12:06:45.068084 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.068053 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" Jul 16 12:06:45.072463 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.072442 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b86a55_a963_41ff_82a1_c528689346a3.slice/crio-29605ed267363b2ebea257a57aff94c8303e6a7c1c3beb61d3a8ed01c673a9b2 WatchSource:0}: Error finding container 29605ed267363b2ebea257a57aff94c8303e6a7c1c3beb61d3a8ed01c673a9b2: Status 404 returned error can't find the container with id 29605ed267363b2ebea257a57aff94c8303e6a7c1c3beb61d3a8ed01c673a9b2 Jul 16 12:06:45.076981 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:06:45.076964 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813c752f_d510_4ccc_8551_317d603d7cf1.slice/crio-ace9077eecd57b55585959e86b2d1b7399420f9946f5774f93229af68eecd19e WatchSource:0}: Error finding container ace9077eecd57b55585959e86b2d1b7399420f9946f5774f93229af68eecd19e: Status 404 returned error can't find the container with id ace9077eecd57b55585959e86b2d1b7399420f9946f5774f93229af68eecd19e Jul 16 12:06:45.343681 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.343575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:45.343844 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.343737 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:45.343844 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.343807 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:46.34378462 +0000 UTC m=+4.037891325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:45.444064 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.443956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:45.444251 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.444119 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:45.444251 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.444188 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:46.444167839 +0000 UTC m=+4.138274542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:45.544793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.544755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:45.544953 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.544931 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:06:45.545057 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.544958 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:06:45.545057 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.544978 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:45.545057 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.545057 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:46.545035633 +0000 UTC m=+4.239142344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:45.789575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.789478 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:45.789731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:45.789638 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:45.798341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.798289 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsmnv" event={"ID":"75b86a55-a963-41ff-82a1-c528689346a3","Type":"ContainerStarted","Data":"29605ed267363b2ebea257a57aff94c8303e6a7c1c3beb61d3a8ed01c673a9b2"} Jul 16 12:06:45.802371 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.802341 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"3b1c7dc6771f7384b860016de531e08f98af97013baa85c148aaaa5c4cd9fbf1"} Jul 16 12:06:45.805238 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.805210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerStarted","Data":"aabce6b5d44d84bcc9c806c98f61a5ac9050275834ef5c267b6e83d1344777ed"} Jul 16 12:06:45.812362 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.812299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kb2qd" event={"ID":"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7","Type":"ContainerStarted","Data":"2e90c03d165206e0967351704d1d282711660107e55ec198e72f2d8a833a6843"} Jul 16 12:06:45.814700 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.814673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbpk4" event={"ID":"1d320534-8242-4791-8292-7fbfca21d331","Type":"ContainerStarted","Data":"541fc78a61d4822952f31e21573177a461f7d8d1f2c0544a7915f0bbae6a6bf8"} Jul 16 12:06:45.820221 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.820194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" event={"ID":"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea","Type":"ContainerStarted","Data":"043cd7104c7459a5d563a4326a223ae555ef64c958543ff1def7b377e3eeb98f"} Jul 16 12:06:45.822365 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.822336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" event={"ID":"813c752f-d510-4ccc-8551-317d603d7cf1","Type":"ContainerStarted","Data":"ace9077eecd57b55585959e86b2d1b7399420f9946f5774f93229af68eecd19e"} Jul 16 12:06:45.824060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.824037 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9r8b" event={"ID":"7628b79f-14bf-4fb4-83e2-b48f5208e310","Type":"ContainerStarted","Data":"ea15ff152db8d3cc7791c5e4b106976e5153c4ad0b77e79409e2de2517911c78"} Jul 16 12:06:45.826190 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.826165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xd8m9" event={"ID":"5c7e88a7-710b-422c-9bae-a64e036e1001","Type":"ContainerStarted","Data":"f2d1d30117b4a151f61c8a5ff3704b0bd72f5dc86eb0ceaf9afa6d6095b80373"} Jul 16 12:06:45.876969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.876219 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:01:45 +0000 UTC" deadline="2028-02-20 11:18:56.972579652 +0000 UTC" Jul 16 12:06:45.876969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:45.876258 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14015h12m11.096326663s" Jul 16 12:06:46.351426 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.351354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:46.351700 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.351584 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:46.351700 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.351669 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:48.351648176 +0000 UTC m=+6.045754876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:46.452805 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.452737 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:46.453013 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.452908 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:46.453013 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.452976 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:48.452955553 +0000 UTC m=+6.147062256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:46.553401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.553357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:46.553634 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.553597 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:06:46.553634 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.553620 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:06:46.553755 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.553638 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:46.553755 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.553719 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:48.553683119 +0000 UTC m=+6.247789818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:46.795188 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.792340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:46.795188 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.792544 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:46.795188 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.792848 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:46.795188 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:46.792978 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:46.877436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.877369 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:01:45 +0000 UTC" deadline="2028-02-14 08:36:52.107529426 +0000 UTC" Jul 16 12:06:46.877436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:46.877431 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13868h30m5.230101706s" Jul 16 12:06:47.788330 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:47.788249 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:47.788505 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:47.788398 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:47.840309 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:47.840268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" event={"ID":"ef1df8b73c3bb3a4c595daeb07adb161","Type":"ContainerStarted","Data":"48ca0d5b319ae75ea2fff3fa248fe9c3827fdd54e814889e0ac4b55ce89206f6"} Jul 16 12:06:48.367129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:48.366741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:48.367701 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.367219 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:48.367701 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.367290 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:52.367268701 +0000 UTC m=+10.061375406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:48.468437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:48.467816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:48.468437 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.468001 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:48.468437 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.468083 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:52.468063904 +0000 UTC m=+10.162170603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:48.569086 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:48.568422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:48.569086 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.568617 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:06:48.569086 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.568637 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:06:48.569086 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.568650 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:48.569086 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.568707 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:06:52.568688297 +0000 UTC m=+10.262795009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:48.791784 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:48.791042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:48.791784 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.791191 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:48.791784 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:48.791581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:48.791784 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:48.791675 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:49.788595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:49.788561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:49.789096 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:49.788708 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:50.788210 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:50.787746 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:50.788210 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:50.787862 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:50.788446 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:50.788245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:50.788446 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:50.788345 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:51.788141 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:51.788109 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:51.788632 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:51.788254 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:51.853662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:51.853567 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef1df8b73c3bb3a4c595daeb07adb161" containerID="48ca0d5b319ae75ea2fff3fa248fe9c3827fdd54e814889e0ac4b55ce89206f6" exitCode=0 Jul 16 12:06:51.853662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:51.853606 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" event={"ID":"ef1df8b73c3bb3a4c595daeb07adb161","Type":"ContainerDied","Data":"48ca0d5b319ae75ea2fff3fa248fe9c3827fdd54e814889e0ac4b55ce89206f6"} Jul 16 12:06:52.399881 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:52.399840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:52.400083 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.400046 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:52.400162 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.400101 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:00.40008722 +0000 UTC m=+18.094193920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:06:52.500521 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:52.500483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:52.500693 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.500634 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:52.500756 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.500708 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:00.500689828 +0000 UTC m=+18.194796538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:06:52.602351 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:52.601670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:52.602351 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.601885 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:06:52.602351 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.601908 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:06:52.602351 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.601921 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:52.602351 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.601977 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:00.601959019 +0000 UTC m=+18.296065733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:06:52.789926 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:52.789322 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:52.789926 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.789442 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:52.789926 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:52.789760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:52.789926 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:52.789845 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:53.788479 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:53.788444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:53.788702 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:53.788593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:54.790026 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:54.788211 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:54.790026 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:54.788347 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:54.790026 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:54.788579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:54.790026 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:54.788693 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:55.788160 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:55.788087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:55.788300 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:55.788235 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:56.788680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:56.788643 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:56.789124 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:56.788645 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:56.789124 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:56.788779 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:56.789124 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:56.788926 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:57.788021 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:57.787975 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:57.788206 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:57.788111 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:06:58.788331 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:58.788295 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:06:58.788766 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:58.788305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:06:58.788766 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:58.788432 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:06:58.788766 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:58.788459 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:06:59.787756 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:06:59.787723 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:06:59.787951 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:06:59.787829 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:00.463541 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:00.463452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:00.463914 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.463577 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:00.463914 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.463634 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:16.463619196 +0000 UTC m=+34.157725893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:00.563783 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:00.563750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:00.563962 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.563925 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:00.564063 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.564005 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:16.563971731 +0000 UTC m=+34.258078428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:00.664407 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:00.664374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:00.664589 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.664531 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:00.664589 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.664558 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:00.664589 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.664570 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:00.664733 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.664636 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:16.664616198 +0000 UTC m=+34.358722901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:00.788600 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:00.788502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:00.788766 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:00.788504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:00.788766 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.788629 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:00.788766 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:00.788690 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:01.788644 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:01.788600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:01.789125 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:01.788743 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:02.789462 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:02.789433 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:02.789462 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:02.789471 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:02.789981 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:02.789527 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:02.789981 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:02.789639 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:03.787978 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:03.787945 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:03.788144 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:03.788069 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:03.872528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:03.872498 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" event={"ID":"7ae3cb7d0f0636b302c0e293270dd787","Type":"ContainerStarted","Data":"6ba7f54a430e09b912cd482cf147b38085840689ee3ad81237ffd011219418e3"} Jul 16 12:07:04.788271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.787859 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:04.788413 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.787920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:04.788413 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:04.788392 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:04.788523 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:04.788489 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:04.877622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.877485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" event={"ID":"813c752f-d510-4ccc-8551-317d603d7cf1","Type":"ContainerStarted","Data":"0c68c52cce2de5a3e52d9e9b7841f9a37e03adc398e929e85993346f5da12a75"} Jul 16 12:07:04.879211 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.879157 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsmnv" event={"ID":"75b86a55-a963-41ff-82a1-c528689346a3","Type":"ContainerStarted","Data":"45afd44dd41a8e13ea6f9a3430118f7a9c6ccdfa9fbcd357909a9b5f82494970"} Jul 16 12:07:04.884472 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.884452 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:07:04.884835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.884807 2583 generic.go:356] "Generic (PLEG): container finished" podID="e4aa1f4d-445c-4753-a5dc-c7c5317fd337" containerID="bd5f770a80e095372530177e7b25d64120932c4ecabdc3c1e0014947213b1c7a" exitCode=1 Jul 16 12:07:04.884962 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.884880 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"ba59b5eb4bd350ccd4f9d7a453dc5365c2763f08f34e44a1d7fb30e290c1df46"} Jul 16 12:07:04.884962 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.884945 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"9ddc2c0984f6bb623f7b0b0952915c74cbc0d0714524627c46136854448fd803"} Jul 16 12:07:04.885114 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.885014 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"9f3e9b9ace9c35312e177097b32c2f6a6c9448254a26e8ac67b26f5ba27b51f7"} Jul 16 12:07:04.885114 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.885064 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"f272dc649247a25eadc6e01531455a27b79507332616274ef3cd3a039b41f134"} Jul 16 12:07:04.885114 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.885111 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerDied","Data":"bd5f770a80e095372530177e7b25d64120932c4ecabdc3c1e0014947213b1c7a"} Jul 16 12:07:04.885262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.885162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"6e59e65e7bb8e83019d0045aad065e95c904f0ba4db1a5b34468d326f2fc8288"} Jul 16 12:07:04.887093 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.887069 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="8e55a21ebd35af389af72586687002aeb7c0461471263589b573518b234724e7" exitCode=0 Jul 16 12:07:04.887176 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.887146 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"8e55a21ebd35af389af72586687002aeb7c0461471263589b573518b234724e7"} Jul 16 12:07:04.891475 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.891426 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kb2qd" event={"ID":"2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7","Type":"ContainerStarted","Data":"c366d6f86ac5f4e4e2e50489b46a01932c836566e1420fce8599623a56c360e5"} Jul 16 12:07:04.893423 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.893399 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbpk4" event={"ID":"1d320534-8242-4791-8292-7fbfca21d331","Type":"ContainerStarted","Data":"afe6bd977daa666d0702cf144b2a1e9c69a556c9591f89247d6f90cf98b56ade"} Jul 16 12:07:04.895127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.895105 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" event={"ID":"fceb94e3-aefe-49d8-8aec-cfef3ee7c1ea","Type":"ContainerStarted","Data":"a45526e29e2f86524ad8d5518988cc0d956d249d0512376b6db3e94ea8e060b6"} Jul 16 12:07:04.897809 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.897786 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" event={"ID":"ef1df8b73c3bb3a4c595daeb07adb161","Type":"ContainerStarted","Data":"e394f74e4f924ceb8b358b5d1a5f8abab8076642e13b47065f4450cd4dfc1210"} Jul 16 12:07:04.919247 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.919189 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dsmnv" podStartSLOduration=6.857530977 podStartE2EDuration="21.919169166s" totalImagesPullingTime="15.061638189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:04.900327503 +0000 UTC m=+22.594434224" watchObservedRunningTime="2026-07-16 12:07:04.919169166 +0000 UTC m=+22.613275886" Jul 16 12:07:04.920180 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.919687 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nnvj8" podStartSLOduration=3.104733183 podStartE2EDuration="21.919675981s" totalImagesPullingTime="18.814942798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:04.919075835 +0000 UTC m=+22.613182581" watchObservedRunningTime="2026-07-16 12:07:04.919675981 +0000 UTC m=+22.613782701" Jul 16 12:07:04.936920 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:04.936869 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kb2qd" podStartSLOduration=3.150005625 podStartE2EDuration="21.936849341s" totalImagesPullingTime="18.786843716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:04.936224173 +0000 UTC m=+22.630330894" watchObservedRunningTime="2026-07-16 12:07:04.936849341 +0000 UTC m=+22.630956064" Jul 16 12:07:05.013553 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.013505 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-22.ec2.internal" podStartSLOduration=21.013488917 podStartE2EDuration="21.013488917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:05.013138616 +0000 UTC m=+22.707245335" watchObservedRunningTime="2026-07-16 12:07:05.013488917 +0000 UTC m=+22.707595635" Jul 16 12:07:05.035138 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.035100 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hbpk4" podStartSLOduration=3.217736615 podStartE2EDuration="22.035083078s" totalImagesPullingTime="18.817346463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:05.034798327 +0000 UTC m=+22.728905046" watchObservedRunningTime="2026-07-16 12:07:05.035083078 +0000 UTC m=+22.729189798" Jul 16 12:07:05.788441 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.788398 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:05.788654 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:05.788543 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:05.902419 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.902382 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9r8b" event={"ID":"7628b79f-14bf-4fb4-83e2-b48f5208e310","Type":"ContainerStarted","Data":"795031f7a10ad0d4d33b115d5762ba34df67de6efa8a0b6f88e5f6a3c9ddc5ee"} Jul 16 12:07:05.904276 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.904244 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xd8m9" event={"ID":"5c7e88a7-710b-422c-9bae-a64e036e1001","Type":"ContainerStarted","Data":"80be27605cac4201729122431300c1f4254c623f9184f2f39017ab08745749ce"} Jul 16 12:07:05.953239 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:05.953188 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z9r8b" podStartSLOduration=4.19246452 podStartE2EDuration="22.953162135s" totalImagesPullingTime="18.760697615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:05.924026129 +0000 UTC m=+23.618132869" watchObservedRunningTime="2026-07-16 12:07:05.953162135 +0000 UTC m=+23.647268857" Jul 16 12:07:06.354680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.354655 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 12:07:06.788290 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.788227 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:06.788496 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.788227 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:06.788496 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:06.788420 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:06.788496 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:06.788479 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:06.838634 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.838531 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T12:07:06.354675244Z","UUID":"1c03f932-66d7-4f4e-9c74-f0891d97ee1d","Handler":null,"Name":"","Endpoint":""} Jul 16 12:07:06.840594 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.840568 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 12:07:06.840741 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.840618 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 12:07:06.908446 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.908412 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" event={"ID":"813c752f-d510-4ccc-8551-317d603d7cf1","Type":"ContainerStarted","Data":"5bfe57e0d5d61693feeb4ea2294322055e877142f66f461c4b8971cde4c3ae3a"} Jul 16 12:07:06.909891 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.909861 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" event={"ID":"7ae3cb7d0f0636b302c0e293270dd787","Type":"ContainerStarted","Data":"ff06f226fa155a39603eaecac2f583b515fd53b559b11456d224b959b4d63338"} Jul 16 12:07:06.937207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:06.937135 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xd8m9" podStartSLOduration=4.00205727 podStartE2EDuration="23.937113885s" totalImagesPullingTime="19.935056615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:05.955098411 +0000 UTC m=+23.649205153" watchObservedRunningTime="2026-07-16 12:07:06.937113885 +0000 UTC m=+24.631220605" Jul 16 12:07:07.788319 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:07.788267 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:07.788512 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:07.788427 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:07.916595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:07.916411 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:07:07.917283 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:07.917246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"a6506fc5e8d89aeb9e6a68d7d9281bbaed5b56cb173914ad47ab0ab3c515aaad"} Jul 16 12:07:08.692073 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.692036 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:07:08.692900 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.692881 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:07:08.711396 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.711345 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-22.ec2.internal" podStartSLOduration=24.711329297 podStartE2EDuration="24.711329297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:06.93802961 +0000 UTC m=+24.632136330" watchObservedRunningTime="2026-07-16 12:07:08.711329297 +0000 UTC m=+26.405436021" Jul 16 12:07:08.788629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.788593 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:08.788793 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:08.788727 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:08.788793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.788763 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:08.788904 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:08.788882 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:08.922285 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.922245 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" event={"ID":"813c752f-d510-4ccc-8551-317d603d7cf1","Type":"ContainerStarted","Data":"bbdd485ce4ee05a49a3bf8632eaf399a7593b5f9aaefeada19757a043407e345"} Jul 16 12:07:08.922840 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.922710 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:07:08.923160 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.923139 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dsmnv" Jul 16 12:07:08.973201 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:08.973069 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b6wsz" podStartSLOduration=3.296253104 podStartE2EDuration="25.973050945s" totalImagesPullingTime="22.676797841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:08.949041304 +0000 UTC m=+26.643148054" watchObservedRunningTime="2026-07-16 12:07:08.973050945 +0000 UTC m=+26.667157676" Jul 16 12:07:09.788522 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:09.788485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:09.788709 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:09.788618 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:10.791265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.791039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:10.791871 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.791055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:10.791871 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:10.791290 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:10.791871 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:10.791386 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:10.929704 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.929676 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:07:10.930053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.930024 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"c021458ea5cd42f2b492d85264606520e7c8acd9d58ec4858c9a5d9eedb647ca"} Jul 16 12:07:10.930371 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.930337 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:10.930535 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.930520 2583 scope.go:117] "RemoveContainer" containerID="bd5f770a80e095372530177e7b25d64120932c4ecabdc3c1e0014947213b1c7a" Jul 16 12:07:10.932059 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.932031 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="4385e9bf1d1a0fc049c42bb6510742dea5c2df03fb6d1ae660bdb3712d59d58a" exitCode=0 Jul 16 12:07:10.932177 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.932110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"4385e9bf1d1a0fc049c42bb6510742dea5c2df03fb6d1ae660bdb3712d59d58a"} Jul 16 12:07:10.947273 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:10.947240 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:11.788094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.788054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:11.788299 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:11.788170 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:11.938268 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.938240 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:07:11.938694 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.938604 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" event={"ID":"e4aa1f4d-445c-4753-a5dc-c7c5317fd337","Type":"ContainerStarted","Data":"1eb9a31011cf7e36fdea3056328501b5d56136d1eea63a473dbbe7bcdf4ea346"} Jul 16 12:07:11.938788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.938766 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:11.938873 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.938826 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:11.954858 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:11.954829 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:12.031798 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.031747 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" podStartSLOduration=10.232114395 podStartE2EDuration="29.031731743s" totalImagesPullingTime="18.799617348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:11.984562835 +0000 UTC m=+29.678669553" watchObservedRunningTime="2026-07-16 12:07:12.031731743 +0000 UTC m=+29.725838461" Jul 16 12:07:12.406455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.406374 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zs6r"] Jul 16 12:07:12.406616 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.406515 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:12.406675 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:12.406630 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:12.420881 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.420853 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gcltf"] Jul 16 12:07:12.421070 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.420973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:12.421139 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:12.421099 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:12.437984 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.437951 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bzpqj"] Jul 16 12:07:12.438132 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.438095 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:12.438215 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:12.438194 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:12.942757 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.942719 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="0f2d23edbef404e5c178471ca8790717ff985b6f8091f0e712adcabbf7840521" exitCode=0 Jul 16 12:07:12.943139 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:12.942801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"0f2d23edbef404e5c178471ca8790717ff985b6f8091f0e712adcabbf7840521"} Jul 16 12:07:14.791205 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:14.791043 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:14.791626 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:14.791055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:14.791626 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:14.791269 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bzpqj" podUID="470371ed-667c-4545-a836-883ce8e71ff5" Jul 16 12:07:14.791626 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:14.791337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcltf" podUID="15f329f3-d691-430e-8061-3e6c8854b383" Jul 16 12:07:14.791626 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:14.791074 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:14.791626 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:14.791458 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:07:16.480069 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.480037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:16.480445 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.480146 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:16.480445 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.480197 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:48.480183465 +0000 UTC m=+66.174290163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:07:16.580588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.580558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:16.580732 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.580669 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:16.580732 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.580716 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret podName:470371ed-667c-4545-a836-883ce8e71ff5 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:48.580704226 +0000 UTC m=+66.274810928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret") pod "global-pull-secret-syncer-bzpqj" (UID: "470371ed-667c-4545-a836-883ce8e71ff5") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:07:16.598298 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.598275 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-22.ec2.internal" event="NodeReady" Jul 16 12:07:16.598421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.598407 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 12:07:16.681076 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.680982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:16.681231 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.681159 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:07:16.681231 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.681177 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:07:16.681231 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.681186 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tv4l8 for pod openshift-network-diagnostics/network-check-target-gcltf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:16.681384 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.681263 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8 podName:15f329f3-d691-430e-8061-3e6c8854b383 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:48.68124468 +0000 UTC m=+66.375351380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4l8" (UniqueName: "kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8") pod "network-check-target-gcltf" (UID: "15f329f3-d691-430e-8061-3e6c8854b383") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:07:16.682413 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.682393 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:07:16.707861 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.707834 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx"] Jul 16 12:07:16.707980 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.707951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.713273 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.713255 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knmbz\"" Jul 16 12:07:16.713371 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.713311 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 12:07:16.714107 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.714081 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 12:07:16.714394 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.714369 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 12:07:16.721725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.721705 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn"] Jul 16 12:07:16.721837 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.721822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.727488 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.727418 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 12:07:16.727812 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.727736 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 12:07:16.727812 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.727744 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 12:07:16.727999 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.727696 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 12:07:16.728123 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.728016 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 12:07:16.728256 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.728219 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 12:07:16.728629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.728614 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 12:07:16.742221 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.742203 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk"] Jul 16 12:07:16.742323 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.742307 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.751166 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.751149 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 12:07:16.751300 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.751284 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 12:07:16.762062 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.762042 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk"] Jul 16 12:07:16.762148 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.762069 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn"] Jul 16 12:07:16.762185 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.762154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:16.778832 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.778810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:07:16.781880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.781863 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ggfxd"] Jul 16 12:07:16.783403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.783388 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cc88g\"" Jul 16 12:07:16.783567 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.783551 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 12:07:16.800749 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.800730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.804090 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.804071 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qt9pz"] Jul 16 12:07:16.804164 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.804148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:16.804199 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.804162 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:16.804248 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.804237 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:16.815656 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.815640 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:16.831409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831391 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 12:07:16.831495 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831450 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:07:16.831495 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831472 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 12:07:16.831495 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831478 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:07:16.831664 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831531 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:07:16.831664 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831593 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:07:16.831761 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831670 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 12:07:16.831761 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831699 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 12:07:16.831917 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.831904 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 12:07:16.832335 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.832319 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qt9pz"] Jul 16 12:07:16.832726 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.832700 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8966\"" Jul 16 12:07:16.833723 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.833697 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w29rm\"" Jul 16 12:07:16.835006 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.834968 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hz6m9\"" Jul 16 12:07:16.835488 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.835427 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pwb6c\"" Jul 16 12:07:16.857449 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.857426 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx"] Jul 16 12:07:16.882542 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882561 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00e1c816-ab61-4a12-8adf-664b88fc974a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:16.882651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32679ff8-a50b-4942-9cc7-9dd8e51df226-klusterlet-config\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd7d\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.882778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882864 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2z48\" (UniqueName: \"kubernetes.io/projected/00e1c816-ab61-4a12-8adf-664b88fc974a-kube-api-access-n2z48\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:16.882968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.882968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.882968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.882975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchjk\" (UniqueName: \"kubernetes.io/projected/7b356e59-3346-4657-9e15-aa94c5abd38c-kube-api-access-gchjk\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b356e59-3346-4657-9e15-aa94c5abd38c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-config-volume\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslf7\" (UniqueName: \"kubernetes.io/projected/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-kube-api-access-jslf7\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883106 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32679ff8-a50b-4942-9cc7-9dd8e51df226-tmp\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.883131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-tmp-dir\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.883344 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.883144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndc4\" (UniqueName: \"kubernetes.io/projected/32679ff8-a50b-4942-9cc7-9dd8e51df226-kube-api-access-zndc4\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.886269 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.886249 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ggfxd"] Jul 16 12:07:16.984272 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.984272 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984222 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.984272 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32679ff8-a50b-4942-9cc7-9dd8e51df226-klusterlet-config\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd7d\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.984360 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.984379 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.984441 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:17.484412858 +0000 UTC m=+35.178519559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2z48\" (UniqueName: \"kubernetes.io/projected/00e1c816-ab61-4a12-8adf-664b88fc974a-kube-api-access-n2z48\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:16.984559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.985022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.985022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.985022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.984951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchjk\" (UniqueName: \"kubernetes.io/projected/7b356e59-3346-4657-9e15-aa94c5abd38c-kube-api-access-gchjk\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.985022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.985217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b356e59-3346-4657-9e15-aa94c5abd38c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.985217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985096 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.985217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-config-volume\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.985217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslf7\" (UniqueName: \"kubernetes.io/projected/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-kube-api-access-jslf7\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.985412 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32679ff8-a50b-4942-9cc7-9dd8e51df226-tmp\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.985660 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.985606 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32679ff8-a50b-4942-9cc7-9dd8e51df226-tmp\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.986343 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-config-volume\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.986780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-tmp-dir\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.987086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndc4\" (UniqueName: \"kubernetes.io/projected/32679ff8-a50b-4942-9cc7-9dd8e51df226-kube-api-access-zndc4\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.987326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de45197d-cd2f-4624-b972-cc76efc7ed86-cert\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.987787 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.989254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.989295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00e1c816-ab61-4a12-8adf-664b88fc974a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.989330 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lgh\" (UniqueName: \"kubernetes.io/projected/de45197d-cd2f-4624-b972-cc76efc7ed86-kube-api-access-45lgh\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.988350 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:16.989472 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:17.489455564 +0000 UTC m=+35.183562262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.989011 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b356e59-3346-4657-9e15-aa94c5abd38c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.990131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.990110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.990775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.990146 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.990775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.990407 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-tmp-dir\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:16.990775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.990763 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.990912 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.990848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.991620 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.991595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32679ff8-a50b-4942-9cc7-9dd8e51df226-klusterlet-config\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:16.992186 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.992166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.993547 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.993518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-ca\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.993630 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.993567 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7b356e59-3346-4657-9e15-aa94c5abd38c-hub\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:16.993782 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.993765 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:16.994588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:16.994568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00e1c816-ab61-4a12-8adf-664b88fc974a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:17.011507 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.011481 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd7d\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:17.020582 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.020556 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2z48\" (UniqueName: \"kubernetes.io/projected/00e1c816-ab61-4a12-8adf-664b88fc974a-kube-api-access-n2z48\") pod \"managed-serviceaccount-addon-agent-5697bd98b8-nv4xk\" (UID: \"00e1c816-ab61-4a12-8adf-664b88fc974a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:17.020685 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.020620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchjk\" (UniqueName: \"kubernetes.io/projected/7b356e59-3346-4657-9e15-aa94c5abd38c-kube-api-access-gchjk\") pod \"cluster-proxy-proxy-agent-5ff47cfbf5-w74hx\" (UID: \"7b356e59-3346-4657-9e15-aa94c5abd38c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:17.022860 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.022842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:17.024072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.024054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndc4\" (UniqueName: \"kubernetes.io/projected/32679ff8-a50b-4942-9cc7-9dd8e51df226-kube-api-access-zndc4\") pod \"klusterlet-addon-workmgr-76bdc9674f-r98vn\" (UID: \"32679ff8-a50b-4942-9cc7-9dd8e51df226\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:17.034238 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.034221 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslf7\" (UniqueName: \"kubernetes.io/projected/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-kube-api-access-jslf7\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:17.039151 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.039136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:07:17.051527 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.051511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:17.069872 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.069846 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:07:17.090301 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.090265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de45197d-cd2f-4624-b972-cc76efc7ed86-cert\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:17.090396 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.090324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lgh\" (UniqueName: \"kubernetes.io/projected/de45197d-cd2f-4624-b972-cc76efc7ed86-kube-api-access-45lgh\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:17.093871 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.093845 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de45197d-cd2f-4624-b972-cc76efc7ed86-cert\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:17.131297 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.131260 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lgh\" (UniqueName: \"kubernetes.io/projected/de45197d-cd2f-4624-b972-cc76efc7ed86-kube-api-access-45lgh\") pod \"ingress-canary-qt9pz\" (UID: \"de45197d-cd2f-4624-b972-cc76efc7ed86\") " pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:17.145055 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.145022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qt9pz" Jul 16 12:07:17.266427 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.266210 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk"] Jul 16 12:07:17.269089 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:17.269062 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e1c816_ab61_4a12_8adf_664b88fc974a.slice/crio-f8b883a813e033b9c30e24fd50e96edd75c82ce721b747b5912c09ebdbd88be8 WatchSource:0}: Error finding container f8b883a813e033b9c30e24fd50e96edd75c82ce721b747b5912c09ebdbd88be8: Status 404 returned error can't find the container with id f8b883a813e033b9c30e24fd50e96edd75c82ce721b747b5912c09ebdbd88be8 Jul 16 12:07:17.271809 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.271788 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn"] Jul 16 12:07:17.280055 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:17.280032 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32679ff8_a50b_4942_9cc7_9dd8e51df226.slice/crio-1f716ea96fb58d0b217ec37179563f061a567f67f2da7bb7a58b24effa193386 WatchSource:0}: Error finding container 1f716ea96fb58d0b217ec37179563f061a567f67f2da7bb7a58b24effa193386: Status 404 returned error can't find the container with id 1f716ea96fb58d0b217ec37179563f061a567f67f2da7bb7a58b24effa193386 Jul 16 12:07:17.303073 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.303047 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx"] Jul 16 12:07:17.306150 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:17.306127 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b356e59_3346_4657_9e15_aa94c5abd38c.slice/crio-577319e9cae5af717588bdae3623f9b9be7a72774bd2841ff23025108a702b65 WatchSource:0}: Error finding container 577319e9cae5af717588bdae3623f9b9be7a72774bd2841ff23025108a702b65: Status 404 returned error can't find the container with id 577319e9cae5af717588bdae3623f9b9be7a72774bd2841ff23025108a702b65 Jul 16 12:07:17.368013 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.367967 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qt9pz"] Jul 16 12:07:17.372449 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:17.372416 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde45197d_cd2f_4624_b972_cc76efc7ed86.slice/crio-1c8298ea1e4e863b08fecbf3cfe7027414c4d8cbae1a5f902ae3e3a1dd12553c WatchSource:0}: Error finding container 1c8298ea1e4e863b08fecbf3cfe7027414c4d8cbae1a5f902ae3e3a1dd12553c: Status 404 returned error can't find the container with id 1c8298ea1e4e863b08fecbf3cfe7027414c4d8cbae1a5f902ae3e3a1dd12553c Jul 16 12:07:17.406535 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.406493 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:17.492329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.492295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.492382 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:17.492459 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:17.492478 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:17.492513 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:17.492543 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:18.492523432 +0000 UTC m=+36.186630132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:17.493081 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:17.492560 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:18.492548168 +0000 UTC m=+36.186654870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:17.592214 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.592123 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:17.954845 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.954807 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qt9pz" event={"ID":"de45197d-cd2f-4624-b972-cc76efc7ed86","Type":"ContainerStarted","Data":"1c8298ea1e4e863b08fecbf3cfe7027414c4d8cbae1a5f902ae3e3a1dd12553c"} Jul 16 12:07:17.955889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.955863 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerStarted","Data":"f8b883a813e033b9c30e24fd50e96edd75c82ce721b747b5912c09ebdbd88be8"} Jul 16 12:07:17.956879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.956855 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerStarted","Data":"1f716ea96fb58d0b217ec37179563f061a567f67f2da7bb7a58b24effa193386"} Jul 16 12:07:17.957797 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:17.957778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerStarted","Data":"577319e9cae5af717588bdae3623f9b9be7a72774bd2841ff23025108a702b65"} Jul 16 12:07:18.500379 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:18.500331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:18.500432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:18.500498 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:18.500523 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:18.500542 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:18.500588 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:20.500568023 +0000 UTC m=+38.194674725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:18.500813 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:18.500608 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:20.500597301 +0000 UTC m=+38.194704000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:18.971456 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:18.971412 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="47c8c152c5c0d63476f1cdca04d12d98c142b1442f2bace77d6dce950f069226" exitCode=0 Jul 16 12:07:18.972388 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:18.972064 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"47c8c152c5c0d63476f1cdca04d12d98c142b1442f2bace77d6dce950f069226"} Jul 16 12:07:20.518938 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:20.518894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:20.519432 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:20.518976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:20.519432 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:20.519144 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:20.519432 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:20.519210 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:24.519190938 +0000 UTC m=+42.213297643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:20.519753 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:20.519732 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:20.519844 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:20.519756 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:20.519915 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:20.519874 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:24.519815448 +0000 UTC m=+42.213922145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:22.913615 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:22.913579 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:23.008196 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:23.008162 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:24.552548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:24.552505 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:24.552597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:24.552668 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:24.552692 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:24.552750 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:32.552736166 +0000 UTC m=+50.246842882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:24.552694 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:24.552969 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:24.552809 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:32.552798356 +0000 UTC m=+50.246905070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:24.919139 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:24.919098 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:25.018101 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:25.018051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:25.909207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:25.909159 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:26.058629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:26.058572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:26.192108 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:26.191976 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:26.824943 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:26.824908 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:28.001096 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.001008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qt9pz" event={"ID":"de45197d-cd2f-4624-b972-cc76efc7ed86","Type":"ContainerStarted","Data":"d8350cfc824b59bad0f16f911238cc360a69aa101736acb396924a14ccb10cca"} Jul 16 12:07:28.003609 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.003584 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="8de060f2978b3c492915e25d1b9164f12cf4e823a3325560de00fd93131cfeaa" exitCode=0 Jul 16 12:07:28.003722 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.003649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"8de060f2978b3c492915e25d1b9164f12cf4e823a3325560de00fd93131cfeaa"} Jul 16 12:07:28.005052 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.005026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerStarted","Data":"3486f3b125b89735a37fa218db0698a00b091e8496ff0a308b234bcebb70957b"} Jul 16 12:07:28.006556 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.006532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerStarted","Data":"83061f153978da74d190052c8343ff0449b05bee5fc2b00763946430ae32ac93"} Jul 16 12:07:28.006637 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.006581 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:28.008522 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.008507 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:07:28.008746 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.008729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerStarted","Data":"5a1dae18df3cf15b7195a6ff852dc651fda048ea890e44cd24c1c2fbfdd17489"} Jul 16 12:07:28.008803 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.008755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerStarted","Data":"840488d6951f3146aaf473bf3313f685f966805a45230c83bbf76aeb1e35ba94"} Jul 16 12:07:28.008803 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.008765 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerStarted","Data":"aa539e88ee5d069049d5d6ff97e70978a7d69487407066813a764454272cc386"} Jul 16 12:07:28.033616 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.033569 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qt9pz" podStartSLOduration=2.011700671 podStartE2EDuration="12.033553688s" totalImagesPullingTime="10.021853017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:28.03314135 +0000 UTC m=+45.727248069" watchObservedRunningTime="2026-07-16 12:07:28.033553688 +0000 UTC m=+45.727660408" Jul 16 12:07:28.085264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.085220 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podStartSLOduration=8.955512286 podStartE2EDuration="19.085205067s" totalImagesPullingTime="10.129692781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:28.084188948 +0000 UTC m=+45.778295668" watchObservedRunningTime="2026-07-16 12:07:28.085205067 +0000 UTC m=+45.779311777" Jul 16 12:07:28.177765 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.177720 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" podStartSLOduration=9.051617104 podStartE2EDuration="19.177703089s" totalImagesPullingTime="10.126085985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:28.175641444 +0000 UTC m=+45.869748162" watchObservedRunningTime="2026-07-16 12:07:28.177703089 +0000 UTC m=+45.871809808" Jul 16 12:07:28.237547 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.237471 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" podStartSLOduration=9.148050689 podStartE2EDuration="19.23745648s" totalImagesPullingTime="10.089405791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:28.236673956 +0000 UTC m=+45.930780674" watchObservedRunningTime="2026-07-16 12:07:28.23745648 +0000 UTC m=+45.931563198" Jul 16 12:07:28.840545 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.840501 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:28.955401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:28.955369 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:29.015482 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:29.015453 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973" containerID="ed037b7aca52b0ee1678a73d37d9eb2a9f5c55e105e084a65d5e5e698f6b09a6" exitCode=0 Jul 16 12:07:29.015859 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:29.015553 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerDied","Data":"ed037b7aca52b0ee1678a73d37d9eb2a9f5c55e105e084a65d5e5e698f6b09a6"} Jul 16 12:07:29.793446 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:29.793415 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:30.022276 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:30.022240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" event={"ID":"b0e7e3ad-4f02-4f06-bb10-d19e1d6e7973","Type":"ContainerStarted","Data":"c589d764dd148893cfedaa7a87d459e77d8d2d2188a278c5f5a77f82781302eb"} Jul 16 12:07:30.061455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:30.061356 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wzhr" podStartSLOduration=8.119512232 podStartE2EDuration="47.061340987s" totalImagesPullingTime="38.941828755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:30.061036713 +0000 UTC m=+47.755143444" watchObservedRunningTime="2026-07-16 12:07:30.061340987 +0000 UTC m=+47.755447706" Jul 16 12:07:30.247058 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:30.247022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:30.565096 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:30.565067 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:32.472091 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:32.472052 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:32.615431 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:32.615398 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:32.615620 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:32.615464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:32.615620 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:32.615560 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:32.615620 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:32.615589 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:32.615620 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:32.615605 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:32.615747 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:32.615666 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:07:48.615650624 +0000 UTC m=+66.309757320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:32.615747 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:32.615682 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:07:48.615675681 +0000 UTC m=+66.309782377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:32.664327 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:32.664288 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:32.885468 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:32.885432 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:33.056524 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:33.056487 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:33.189058 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:33.188948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:33.327998 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:33.327962 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:33.755554 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:33.755526 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:34.045508 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:34.045425 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:34.194001 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:34.193951 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:34.313406 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:34.313313 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:37.618482 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:37.618448 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:42.453224 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:42.453187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:42.554726 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:42.554693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:43.954969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:43.954935 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rd4b2" Jul 16 12:07:44.251310 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:44.251217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:44.348048 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:44.348016 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:44.958787 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:44.958748 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:45.127211 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:45.127168 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:48.546659 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.546603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:07:48.559672 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.559647 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:07:48.567211 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.567189 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:07:48.567274 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.567260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:08:52.567238284 +0000 UTC m=+130.261344993 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : secret "metrics-daemon-secret" not found Jul 16 12:07:48.647455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.647419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:07:48.647595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.647477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:07:48.647595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.647501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:48.647595 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.647569 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:07:48.647595 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.647587 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:07:48.647719 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.647634 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:07:48.647719 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.647637 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:08:20.647621639 +0000 UTC m=+98.341728338 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:07:48.647719 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:07:48.647698 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:08:20.647683535 +0000 UTC m=+98.341790245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:07:48.655711 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.655690 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:07:48.660914 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.660895 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/470371ed-667c-4545-a836-883ce8e71ff5-original-pull-secret\") pod \"global-pull-secret-syncer-bzpqj\" (UID: \"470371ed-667c-4545-a836-883ce8e71ff5\") " pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:48.748264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.748215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:48.751399 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.751377 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:07:48.764363 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.764323 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:07:48.771723 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.771702 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4l8\" (UniqueName: \"kubernetes.io/projected/15f329f3-d691-430e-8061-3e6c8854b383-kube-api-access-tv4l8\") pod \"network-check-target-gcltf\" (UID: \"15f329f3-d691-430e-8061-3e6c8854b383\") " pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:48.927699 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.927669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bzpqj" Jul 16 12:07:48.947915 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.947885 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hz6m9\"" Jul 16 12:07:48.951062 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:48.951036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:49.100863 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:49.100837 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470371ed_667c_4545_a836_883ce8e71ff5.slice/crio-2d307f16d7675ca5893b9ea9f8f0be4ab58b7ae77d2e159d6c21fc405c121fa2 WatchSource:0}: Error finding container 2d307f16d7675ca5893b9ea9f8f0be4ab58b7ae77d2e159d6c21fc405c121fa2: Status 404 returned error can't find the container with id 2d307f16d7675ca5893b9ea9f8f0be4ab58b7ae77d2e159d6c21fc405c121fa2 Jul 16 12:07:49.110379 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:49.110357 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bzpqj"] Jul 16 12:07:49.116487 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:49.116467 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gcltf"] Jul 16 12:07:49.121449 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:07:49.121422 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f329f3_d691_430e_8061_3e6c8854b383.slice/crio-d4b9d330cc68041e38f93670dcdc173293b8a34c234e1e70cdb79ee69fe03f5f WatchSource:0}: Error finding container d4b9d330cc68041e38f93670dcdc173293b8a34c234e1e70cdb79ee69fe03f5f: Status 404 returned error can't find the container with id d4b9d330cc68041e38f93670dcdc173293b8a34c234e1e70cdb79ee69fe03f5f Jul 16 12:07:50.086909 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:50.086866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bzpqj" event={"ID":"470371ed-667c-4545-a836-883ce8e71ff5","Type":"ContainerStarted","Data":"2d307f16d7675ca5893b9ea9f8f0be4ab58b7ae77d2e159d6c21fc405c121fa2"} Jul 16 12:07:50.088022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:50.087984 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gcltf" event={"ID":"15f329f3-d691-430e-8061-3e6c8854b383","Type":"ContainerStarted","Data":"d4b9d330cc68041e38f93670dcdc173293b8a34c234e1e70cdb79ee69fe03f5f"} Jul 16 12:07:54.893838 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:54.893793 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:55.010963 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.010932 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:07:55.109101 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.109001 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gcltf" event={"ID":"15f329f3-d691-430e-8061-3e6c8854b383","Type":"ContainerStarted","Data":"c3b92c0a19fd1f6ab785adab7ce821178b56079ae45681be4d0ade6d52783827"} Jul 16 12:07:55.109268 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.109158 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:07:55.110467 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.110445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bzpqj" event={"ID":"470371ed-667c-4545-a836-883ce8e71ff5","Type":"ContainerStarted","Data":"4aa876a6ff30b98107759d5452d1b706a2124d627e31f1727d9644aefa40e2e2"} Jul 16 12:07:55.136854 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.136795 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gcltf" podStartSLOduration=66.396383563 podStartE2EDuration="1m12.136778222s" totalImagesPullingTime="5.740394659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:55.136649106 +0000 UTC m=+72.830755825" watchObservedRunningTime="2026-07-16 12:07:55.136778222 +0000 UTC m=+72.830884941" Jul 16 12:07:55.169215 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.169154 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bzpqj" podStartSLOduration=66.41774013 podStartE2EDuration="1m12.16913451s" totalImagesPullingTime="5.75139438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:07:55.168839577 +0000 UTC m=+72.862946296" watchObservedRunningTime="2026-07-16 12:07:55.16913451 +0000 UTC m=+72.863241228" Jul 16 12:07:55.965114 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:07:55.965074 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" limit=1024 Jul 16 12:08:20.689120 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:08:20.689078 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:08:20.689120 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:08:20.689131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:08:20.689628 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:20.689233 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:08:20.689628 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:20.689235 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:08:20.689628 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:20.689255 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:08:20.689628 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:20.689283 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:09:24.689270861 +0000 UTC m=+162.383377558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:08:20.689628 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:20.689305 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:09:24.689293264 +0000 UTC m=+162.383399961 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:08:26.116674 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:08:26.116641 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gcltf" Jul 16 12:08:52.625627 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:08:52.625577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:08:52.626166 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:52.625747 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:08:52.626166 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:08:52.625828 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs podName:3aa69c1f-fde7-408e-9175-aff918f86270 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:54.625809792 +0000 UTC m=+252.319916488 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs") pod "network-metrics-daemon-4zs6r" (UID: "3aa69c1f-fde7-408e-9175-aff918f86270") : secret "metrics-daemon-secret" not found Jul 16 12:09:19.718769 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:19.718722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" Jul 16 12:09:19.822669 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:19.822620 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ggfxd" podUID="e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3" Jul 16 12:09:19.833893 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:19.833866 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4zs6r" podUID="3aa69c1f-fde7-408e-9175-aff918f86270" Jul 16 12:09:20.353939 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:20.353908 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:09:20.354116 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:20.353908 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:09:24.761325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:24.761277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:24.761336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:24.761432 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:24.761442 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:24.761452 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5854b8c858-7t9j4: secret "image-registry-tls" not found Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:24.761495 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls podName:e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3 nodeName:}" failed. No retries permitted until 2026-07-16 12:11:26.761482126 +0000 UTC m=+284.455588823 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls") pod "dns-default-ggfxd" (UID: "e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3") : secret "dns-default-metrics-tls" not found Jul 16 12:09:24.761731 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:09:24.761507 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls podName:388a0ec4-564c-4a5f-8b4d-487e8ef9742b nodeName:}" failed. No retries permitted until 2026-07-16 12:11:26.761501186 +0000 UTC m=+284.455607884 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls") pod "image-registry-5854b8c858-7t9j4" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b") : secret "image-registry-tls" not found Jul 16 12:09:28.007518 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.007470 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podUID="32679ff8-a50b-4942-9cc7-9dd8e51df226" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused" Jul 16 12:09:28.378821 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.378790 2583 generic.go:356] "Generic (PLEG): container finished" podID="00e1c816-ab61-4a12-8adf-664b88fc974a" containerID="3486f3b125b89735a37fa218db0698a00b091e8496ff0a308b234bcebb70957b" exitCode=255 Jul 16 12:09:28.379017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.378865 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerDied","Data":"3486f3b125b89735a37fa218db0698a00b091e8496ff0a308b234bcebb70957b"} Jul 16 12:09:28.379238 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.379219 2583 scope.go:117] "RemoveContainer" containerID="3486f3b125b89735a37fa218db0698a00b091e8496ff0a308b234bcebb70957b" Jul 16 12:09:28.380565 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.380410 2583 generic.go:356] "Generic (PLEG): container finished" podID="32679ff8-a50b-4942-9cc7-9dd8e51df226" containerID="83061f153978da74d190052c8343ff0449b05bee5fc2b00763946430ae32ac93" exitCode=1 Jul 16 12:09:28.380565 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.380451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerDied","Data":"83061f153978da74d190052c8343ff0449b05bee5fc2b00763946430ae32ac93"} Jul 16 12:09:28.380727 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:28.380714 2583 scope.go:117] "RemoveContainer" containerID="83061f153978da74d190052c8343ff0449b05bee5fc2b00763946430ae32ac93" Jul 16 12:09:29.385292 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:29.385255 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerStarted","Data":"ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e"} Jul 16 12:09:29.385732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:29.385543 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:09:29.386170 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:29.386152 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:09:29.386922 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:29.386902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerStarted","Data":"71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3"} Jul 16 12:09:32.789808 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:09:32.789764 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:10:43.788543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.788514 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jj2mx"] Jul 16 12:10:43.790523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.790506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:43.795366 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.795340 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fngt\"" Jul 16 12:10:43.795523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.795343 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 12:10:43.795523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.795383 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 12:10:43.796498 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.796190 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 12:10:43.797615 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.797596 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 12:10:43.811543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.811518 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jj2mx"] Jul 16 12:10:43.933006 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.932958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:43.933151 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.933015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpd8\" (UniqueName: \"kubernetes.io/projected/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-api-access-qzpd8\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:43.933151 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.933099 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4396a3f0-4278-4947-a03a-eca9e09dff08-data-volume\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:43.933151 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.933142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:43.933265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:43.933179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4396a3f0-4278-4947-a03a-eca9e09dff08-crio-socket\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpd8\" (UniqueName: \"kubernetes.io/projected/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-api-access-qzpd8\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4396a3f0-4278-4947-a03a-eca9e09dff08-data-volume\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4396a3f0-4278-4947-a03a-eca9e09dff08-crio-socket\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.033781 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.033745 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.034036 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.033817 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:44.533801668 +0000 UTC m=+242.227908364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.034036 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.033826 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:44.034036 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.033905 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:44.533886062 +0000 UTC m=+242.227992773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : configmap "kube-rbac-proxy" not found Jul 16 12:10:44.034036 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.033833 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4396a3f0-4278-4947-a03a-eca9e09dff08-crio-socket\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.034172 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.034058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4396a3f0-4278-4947-a03a-eca9e09dff08-data-volume\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.046662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.046613 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpd8\" (UniqueName: \"kubernetes.io/projected/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-api-access-qzpd8\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.537910 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.537880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.538116 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:44.537928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:44.538116 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.538028 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:44.538116 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.538066 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:44.538116 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.538111 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:45.538098136 +0000 UTC m=+243.232204838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : configmap "kube-rbac-proxy" not found Jul 16 12:10:44.538340 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:44.538157 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:45.538135843 +0000 UTC m=+243.232242540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:45.544711 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:45.544679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:45.545127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:45.544741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:45.545127 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:45.544841 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:45.545127 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:45.544909 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:47.544894949 +0000 UTC m=+245.239001649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:45.545127 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:45.544841 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:45.545127 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:45.544972 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:47.544961429 +0000 UTC m=+245.239068128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : configmap "kube-rbac-proxy" not found Jul 16 12:10:45.759403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:45.759377 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbpk4_1d320534-8242-4791-8292-7fbfca21d331/dns-node-resolver/0.log" Jul 16 12:10:46.745908 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:46.745881 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kb2qd_2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7/node-ca/0.log" Jul 16 12:10:47.552103 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:47.552077 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qt9pz_de45197d-cd2f-4624-b972-cc76efc7ed86/serve-healthcheck-canary/0.log" Jul 16 12:10:47.559565 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:47.559545 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:47.559627 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:47.559590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:47.559688 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:47.559673 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:47.559722 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:47.559681 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:47.559759 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:47.559728 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:51.559714073 +0000 UTC m=+249.253820770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : configmap "kube-rbac-proxy" not found Jul 16 12:10:47.559759 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:47.559741 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:51.559735124 +0000 UTC m=+249.253841820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:51.590908 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:51.590865 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:51.591342 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:51.590943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:51.591342 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:51.590978 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:10:51.591342 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:51.591051 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:10:51.591342 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:51.591063 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:59.591048505 +0000 UTC m=+257.285155205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : configmap "kube-rbac-proxy" not found Jul 16 12:10:51.591342 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:10:51.591092 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls podName:4396a3f0-4278-4947-a03a-eca9e09dff08 nodeName:}" failed. No retries permitted until 2026-07-16 12:10:59.591081667 +0000 UTC m=+257.285188364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jj2mx" (UID: "4396a3f0-4278-4947-a03a-eca9e09dff08") : secret "insights-runtime-extractor-tls" not found Jul 16 12:10:54.715139 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:54.715088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:10:54.717479 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:54.717443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aa69c1f-fde7-408e-9175-aff918f86270-metrics-certs\") pod \"network-metrics-daemon-4zs6r\" (UID: \"3aa69c1f-fde7-408e-9175-aff918f86270\") " pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:10:54.996125 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:54.996035 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w29rm\"" Jul 16 12:10:55.001594 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:55.001574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zs6r" Jul 16 12:10:55.136795 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:55.136764 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zs6r"] Jul 16 12:10:55.139830 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:10:55.139809 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa69c1f_fde7_408e_9175_aff918f86270.slice/crio-007e92c75eefcdda58cfcee4f17c6e201def64deaef28dbe1cc7de6e1e50f3c1 WatchSource:0}: Error finding container 007e92c75eefcdda58cfcee4f17c6e201def64deaef28dbe1cc7de6e1e50f3c1: Status 404 returned error can't find the container with id 007e92c75eefcdda58cfcee4f17c6e201def64deaef28dbe1cc7de6e1e50f3c1 Jul 16 12:10:55.635617 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:55.635578 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zs6r" event={"ID":"3aa69c1f-fde7-408e-9175-aff918f86270","Type":"ContainerStarted","Data":"007e92c75eefcdda58cfcee4f17c6e201def64deaef28dbe1cc7de6e1e50f3c1"} Jul 16 12:10:56.640793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:56.640756 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zs6r" event={"ID":"3aa69c1f-fde7-408e-9175-aff918f86270","Type":"ContainerStarted","Data":"ae71357674700be4b99a4638ea0f55dd253681ee094147c6df1779a17f822f6b"} Jul 16 12:10:56.641209 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:56.640815 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zs6r" event={"ID":"3aa69c1f-fde7-408e-9175-aff918f86270","Type":"ContainerStarted","Data":"ea723e2b32fd33db741ad1e4e53000095d504bc5e843dd6ab634844c376f8bdd"} Jul 16 12:10:56.668083 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:56.668032 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4zs6r" podStartSLOduration=252.794479169 podStartE2EDuration="4m13.66801476s" totalImagesPullingTime="873.535591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:10:56.667571301 +0000 UTC m=+254.361678053" watchObservedRunningTime="2026-07-16 12:10:56.66801476 +0000 UTC m=+254.362121475" Jul 16 12:10:59.656838 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.656805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:59.657204 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.656894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:59.657343 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.657324 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4396a3f0-4278-4947-a03a-eca9e09dff08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:59.659121 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.659104 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4396a3f0-4278-4947-a03a-eca9e09dff08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jj2mx\" (UID: \"4396a3f0-4278-4947-a03a-eca9e09dff08\") " pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:59.699425 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.699391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jj2mx" Jul 16 12:10:59.847149 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:10:59.847114 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jj2mx"] Jul 16 12:10:59.848224 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:10:59.848194 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4396a3f0_4278_4947_a03a_eca9e09dff08.slice/crio-058c0edcff2f87acb2a44c33c5a7ed4ce2e628e853cc51580d48fdb4ece50642 WatchSource:0}: Error finding container 058c0edcff2f87acb2a44c33c5a7ed4ce2e628e853cc51580d48fdb4ece50642: Status 404 returned error can't find the container with id 058c0edcff2f87acb2a44c33c5a7ed4ce2e628e853cc51580d48fdb4ece50642 Jul 16 12:11:00.656879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:00.656783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj2mx" event={"ID":"4396a3f0-4278-4947-a03a-eca9e09dff08","Type":"ContainerStarted","Data":"058c0edcff2f87acb2a44c33c5a7ed4ce2e628e853cc51580d48fdb4ece50642"} Jul 16 12:11:05.674433 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:05.674392 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj2mx" event={"ID":"4396a3f0-4278-4947-a03a-eca9e09dff08","Type":"ContainerStarted","Data":"1b19b94f8a82efe2940b373f3b695e55304d5e97e569b44c28f0674b3c584b64"} Jul 16 12:11:08.688908 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:08.688867 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj2mx" event={"ID":"4396a3f0-4278-4947-a03a-eca9e09dff08","Type":"ContainerStarted","Data":"620b091c8d227a6bb36eec5a503c1c26ace40ce17d1bd7d6c4e105524d4d2a51"} Jul 16 12:11:11.700444 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:11.700411 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jj2mx" event={"ID":"4396a3f0-4278-4947-a03a-eca9e09dff08","Type":"ContainerStarted","Data":"f7ff7e35182ac70854e5724d79f584cde05a53a5747232bb73c48cc40641fa25"} Jul 16 12:11:11.743588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:11.743533 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jj2mx" podStartSLOduration=17.074816531 podStartE2EDuration="28.743517548s" totalImagesPullingTime="11.668701017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:10:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:11.740849346 +0000 UTC m=+269.434956065" watchObservedRunningTime="2026-07-16 12:11:11.743517548 +0000 UTC m=+269.437624266" Jul 16 12:11:16.684941 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.684903 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xnl6v"] Jul 16 12:11:16.687187 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.687170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.697260 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.697237 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 12:11:16.697406 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.697321 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wcl2h\"" Jul 16 12:11:16.698663 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.698646 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 12:11:16.704981 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.704961 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 12:11:16.707803 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.707790 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 12:11:16.708308 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.708292 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 12:11:16.709586 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.709568 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 12:11:16.785880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.785842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-sys\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.785880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.785880 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-wtmp\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.785898 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-metrics-client-ca\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.785949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-root\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.785983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5sh\" (UniqueName: \"kubernetes.io/projected/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-kube-api-access-2p5sh\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.786047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.786078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-textfile\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.786096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-tls\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.786119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.786115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.886652 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886616 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-textfile\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.886652 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-tls\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-sys\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-wtmp\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-metrics-client-ca\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.886969 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-sys\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-root\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-textfile\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5sh\" (UniqueName: \"kubernetes.io/projected/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-kube-api-access-2p5sh\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-root\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887176 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-wtmp\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887693 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887585 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-metrics-client-ca\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.887693 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.887672 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.889131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.889111 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-tls\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.889219 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.889198 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.929880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.929846 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5sh\" (UniqueName: \"kubernetes.io/projected/8b1abf52-0d38-4c7f-b0b6-199a57a787fd-kube-api-access-2p5sh\") pod \"node-exporter-xnl6v\" (UID: \"8b1abf52-0d38-4c7f-b0b6-199a57a787fd\") " pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:16.996206 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:16.996118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xnl6v" Jul 16 12:11:17.006242 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:11:17.006212 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1abf52_0d38_4c7f_b0b6_199a57a787fd.slice/crio-3cc8bcf4a6c3c436235f5be8c7b552eb8027c16f440661c092a63d69cb8ca82b WatchSource:0}: Error finding container 3cc8bcf4a6c3c436235f5be8c7b552eb8027c16f440661c092a63d69cb8ca82b: Status 404 returned error can't find the container with id 3cc8bcf4a6c3c436235f5be8c7b552eb8027c16f440661c092a63d69cb8ca82b Jul 16 12:11:17.718648 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:17.718614 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnl6v" event={"ID":"8b1abf52-0d38-4c7f-b0b6-199a57a787fd","Type":"ContainerStarted","Data":"3cc8bcf4a6c3c436235f5be8c7b552eb8027c16f440661c092a63d69cb8ca82b"} Jul 16 12:11:18.723091 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:18.723055 2583 generic.go:356] "Generic (PLEG): container finished" podID="8b1abf52-0d38-4c7f-b0b6-199a57a787fd" containerID="1e160f42efd05ce92b98631c5c6e1ae1a0262c8859e32766e20ab17c2307cfb0" exitCode=0 Jul 16 12:11:18.723574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:18.723109 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnl6v" event={"ID":"8b1abf52-0d38-4c7f-b0b6-199a57a787fd","Type":"ContainerDied","Data":"1e160f42efd05ce92b98631c5c6e1ae1a0262c8859e32766e20ab17c2307cfb0"} Jul 16 12:11:19.728304 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:19.728273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnl6v" event={"ID":"8b1abf52-0d38-4c7f-b0b6-199a57a787fd","Type":"ContainerStarted","Data":"f366d919ed352dc7216b6324a6266539bb40ee150839a27065f9286cf5535364"} Jul 16 12:11:19.728667 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:19.728360 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnl6v" event={"ID":"8b1abf52-0d38-4c7f-b0b6-199a57a787fd","Type":"ContainerStarted","Data":"ccf3f6d4abc00385a1f530d4a84b8a17c1468f52177239d3c383ac2159fb022f"} Jul 16 12:11:19.767958 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:19.767911 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xnl6v" podStartSLOduration=2.777000883 podStartE2EDuration="3.767895034s" totalImagesPullingTime="929.404817ms" totalInitContainerRuntime="61.489334ms" isStatefulPod=true podCreationTimestamp="2026-07-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:19.767631254 +0000 UTC m=+277.461737973" watchObservedRunningTime="2026-07-16 12:11:19.767895034 +0000 UTC m=+277.462001753" Jul 16 12:11:23.355179 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:23.355136 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ggfxd" podUID="e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3" Jul 16 12:11:23.355179 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:23.355137 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" Jul 16 12:11:23.740940 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:23.740906 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:23.741154 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:23.740918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:26.857225 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:26.857183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:26.857718 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:26.857245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:26.859622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:26.859595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3-metrics-tls\") pod \"dns-default-ggfxd\" (UID: \"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3\") " pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:26.859740 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:26.859651 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"image-registry-5854b8c858-7t9j4\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:27.040858 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.040817 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" podUID="7b356e59-3346-4657-9e15-aa94c5abd38c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 12:11:27.044652 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.044634 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8966\"" Jul 16 12:11:27.045747 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.045734 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knmbz\"" Jul 16 12:11:27.052380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.052362 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:27.052498 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.052461 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:27.209516 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.209479 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:11:27.215567 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:11:27.215540 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388a0ec4_564c_4a5f_8b4d_487e8ef9742b.slice/crio-0bd82559d957592f1b3c1a32176a7dde9bb0b0f52996978e51cc95fc35c328d2 WatchSource:0}: Error finding container 0bd82559d957592f1b3c1a32176a7dde9bb0b0f52996978e51cc95fc35c328d2: Status 404 returned error can't find the container with id 0bd82559d957592f1b3c1a32176a7dde9bb0b0f52996978e51cc95fc35c328d2 Jul 16 12:11:27.229682 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.229657 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ggfxd"] Jul 16 12:11:27.232142 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:11:27.232116 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e196ea_dfbd_41e9_9b2d_3b18e000c6e3.slice/crio-9727ccb56479ade23d28f409aba167993f663214129c78f0ab4d942698226570 WatchSource:0}: Error finding container 9727ccb56479ade23d28f409aba167993f663214129c78f0ab4d942698226570: Status 404 returned error can't find the container with id 9727ccb56479ade23d28f409aba167993f663214129c78f0ab4d942698226570 Jul 16 12:11:27.754977 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.754937 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggfxd" event={"ID":"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3","Type":"ContainerStarted","Data":"9727ccb56479ade23d28f409aba167993f663214129c78f0ab4d942698226570"} Jul 16 12:11:27.756265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.756239 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" event={"ID":"388a0ec4-564c-4a5f-8b4d-487e8ef9742b","Type":"ContainerStarted","Data":"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c"} Jul 16 12:11:27.756408 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.756285 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" event={"ID":"388a0ec4-564c-4a5f-8b4d-487e8ef9742b","Type":"ContainerStarted","Data":"0bd82559d957592f1b3c1a32176a7dde9bb0b0f52996978e51cc95fc35c328d2"} Jul 16 12:11:27.756408 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:27.756361 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:28.761833 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.761754 2583 generic.go:356] "Generic (PLEG): container finished" podID="00e1c816-ab61-4a12-8adf-664b88fc974a" containerID="71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3" exitCode=255 Jul 16 12:11:28.762289 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.761832 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerDied","Data":"71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3"} Jul 16 12:11:28.762289 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.761898 2583 scope.go:117] "RemoveContainer" containerID="3486f3b125b89735a37fa218db0698a00b091e8496ff0a308b234bcebb70957b" Jul 16 12:11:28.762289 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.762250 2583 scope.go:117] "RemoveContainer" containerID="71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3" Jul 16 12:11:28.763007 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:28.762954 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-5697bd98b8-nv4xk_open-cluster-management-agent-addon(00e1c816-ab61-4a12-8adf-664b88fc974a)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" podUID="00e1c816-ab61-4a12-8adf-664b88fc974a" Jul 16 12:11:28.763961 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.763940 2583 generic.go:356] "Generic (PLEG): container finished" podID="32679ff8-a50b-4942-9cc7-9dd8e51df226" containerID="ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e" exitCode=1 Jul 16 12:11:28.764091 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.764022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerDied","Data":"ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e"} Jul 16 12:11:28.764426 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.764410 2583 scope.go:117] "RemoveContainer" containerID="ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e" Jul 16 12:11:28.764600 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:28.764577 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-76bdc9674f-r98vn_open-cluster-management-agent-addon(32679ff8-a50b-4942-9cc7-9dd8e51df226)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podUID="32679ff8-a50b-4942-9cc7-9dd8e51df226" Jul 16 12:11:28.800351 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:28.800294 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" podStartSLOduration=285.800277949 podStartE2EDuration="4m45.800277949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:27.79052988 +0000 UTC m=+285.484636610" watchObservedRunningTime="2026-07-16 12:11:28.800277949 +0000 UTC m=+286.494384667" Jul 16 12:11:29.144089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.144063 2583 scope.go:117] "RemoveContainer" containerID="83061f153978da74d190052c8343ff0449b05bee5fc2b00763946430ae32ac93" Jul 16 12:11:29.386089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.386061 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:11:29.769244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.769216 2583 scope.go:117] "RemoveContainer" containerID="ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e" Jul 16 12:11:29.769664 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:29.769387 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-76bdc9674f-r98vn_open-cluster-management-agent-addon(32679ff8-a50b-4942-9cc7-9dd8e51df226)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podUID="32679ff8-a50b-4942-9cc7-9dd8e51df226" Jul 16 12:11:29.772093 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.772069 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggfxd" event={"ID":"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3","Type":"ContainerStarted","Data":"27af1d6af4dbb4630fcb7586ff46de1f54349d1918c769eec58e59a16925db8d"} Jul 16 12:11:29.772204 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.772114 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggfxd" event={"ID":"e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3","Type":"ContainerStarted","Data":"69dd579931762b23d02ad7e58e2e83ae2e818ea7a6ace38ced3ba497311847c5"} Jul 16 12:11:29.772204 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.772194 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:29.823301 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:29.823236 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ggfxd" podStartSLOduration=251.869105984 podStartE2EDuration="4m13.823212649s" totalImagesPullingTime="1.954106665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:11:29.822558424 +0000 UTC m=+287.516665169" watchObservedRunningTime="2026-07-16 12:11:29.823212649 +0000 UTC m=+287.517319369" Jul 16 12:11:37.040380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:37.040343 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" podUID="7b356e59-3346-4657-9e15-aa94c5abd38c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 12:11:37.052473 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:37.052443 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:11:37.052897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:37.052876 2583 scope.go:117] "RemoveContainer" containerID="ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e" Jul 16 12:11:37.053128 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:37.053101 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-76bdc9674f-r98vn_open-cluster-management-agent-addon(32679ff8-a50b-4942-9cc7-9dd8e51df226)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podUID="32679ff8-a50b-4942-9cc7-9dd8e51df226" Jul 16 12:11:37.070721 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:37.070700 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" Jul 16 12:11:37.071018 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:37.070984 2583 scope.go:117] "RemoveContainer" containerID="71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3" Jul 16 12:11:37.071164 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:11:37.071147 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-5697bd98b8-nv4xk_open-cluster-management-agent-addon(00e1c816-ab61-4a12-8adf-664b88fc974a)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" podUID="00e1c816-ab61-4a12-8adf-664b88fc974a" Jul 16 12:11:39.777598 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:39.777566 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ggfxd" Jul 16 12:11:40.303522 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:40.303489 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:11:42.729008 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:42.728961 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:11:42.730768 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:42.730746 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:11:42.735158 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:42.735139 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 12:11:47.041084 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.041037 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" podUID="7b356e59-3346-4657-9e15-aa94c5abd38c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 12:11:47.042590 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.041109 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" Jul 16 12:11:47.042590 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.041674 2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"5a1dae18df3cf15b7195a6ff852dc651fda048ea890e44cd24c1c2fbfdd17489"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 16 12:11:47.042590 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.041724 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" podUID="7b356e59-3346-4657-9e15-aa94c5abd38c" containerName="service-proxy" containerID="cri-o://5a1dae18df3cf15b7195a6ff852dc651fda048ea890e44cd24c1c2fbfdd17489" gracePeriod=30 Jul 16 12:11:47.830462 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.830418 2583 generic.go:356] "Generic (PLEG): container finished" podID="7b356e59-3346-4657-9e15-aa94c5abd38c" containerID="5a1dae18df3cf15b7195a6ff852dc651fda048ea890e44cd24c1c2fbfdd17489" exitCode=2 Jul 16 12:11:47.830636 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.830486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerDied","Data":"5a1dae18df3cf15b7195a6ff852dc651fda048ea890e44cd24c1c2fbfdd17489"} Jul 16 12:11:47.830636 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:47.830530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ff47cfbf5-w74hx" event={"ID":"7b356e59-3346-4657-9e15-aa94c5abd38c","Type":"ContainerStarted","Data":"94e186c617fcd77b91e92b0685107f7d34e8ee34e54e9d6df6f516b3fdb55be1"} Jul 16 12:11:49.789098 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:49.789065 2583 scope.go:117] "RemoveContainer" containerID="71dfd4b28bd29cb3a63acc5a52315371f58ff3a3c648eae7c073c7a6f2de85e3" Jul 16 12:11:50.309033 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:50.309007 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:11:50.841598 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:50.841564 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5697bd98b8-nv4xk" event={"ID":"00e1c816-ab61-4a12-8adf-664b88fc974a","Type":"ContainerStarted","Data":"2afc93626f2f5433935ca62fa3663c0e2d62ea092d16299e40d592031d8ef9a1"} Jul 16 12:11:51.788689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:51.788664 2583 scope.go:117] "RemoveContainer" containerID="ba349c178f5eb990aeb00887e7bb21e9944c218ede1bd4e6116be9c7a81a2c1e" Jul 16 12:11:51.845756 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:51.845734 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" event={"ID":"32679ff8-a50b-4942-9cc7-9dd8e51df226","Type":"ContainerStarted","Data":"78bfaae17dbdb8d306f33deae783d16560ceed1781bac6ff7ad37e0bac751fd2"} Jul 16 12:11:51.846116 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:51.846014 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:11:51.847143 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:51.847120 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" podUID="32679ff8-a50b-4942-9cc7-9dd8e51df226" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused" Jul 16 12:11:52.849738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:11:52.849711 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76bdc9674f-r98vn" Jul 16 12:12:05.322764 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.322722 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" containerName="registry" containerID="cri-o://c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c" gracePeriod=30 Jul 16 12:12:05.568299 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.568276 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:12:05.646172 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646151 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646290 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646183 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npd7d\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646290 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646209 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646290 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646225 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646290 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646257 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646508 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646308 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646508 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646345 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646508 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646389 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates\") pod \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\" (UID: \"388a0ec4-564c-4a5f-8b4d-487e8ef9742b\") " Jul 16 12:12:05.646666 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646630 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:05.646914 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.646890 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:12:05.648542 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.648506 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:12:05.648644 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.648539 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:05.648842 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.648818 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:12:05.648842 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.648830 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d" (OuterVolumeSpecName: "kube-api-access-npd7d") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "kube-api-access-npd7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:12:05.649119 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.649090 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:12:05.655248 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.655224 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "388a0ec4-564c-4a5f-8b4d-487e8ef9742b" (UID: "388a0ec4-564c-4a5f-8b4d-487e8ef9742b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:12:05.746828 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746806 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-bound-sa-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746828 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746824 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-certificates\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746833 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-trusted-ca\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746843 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npd7d\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-kube-api-access-npd7d\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746851 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-registry-tls\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746859 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-installation-pull-secrets\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746867 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-ca-trust-extracted\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.746942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.746876 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/388a0ec4-564c-4a5f-8b4d-487e8ef9742b-image-registry-private-configuration\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:12:05.893394 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.893367 2583 generic.go:356] "Generic (PLEG): container finished" podID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" containerID="c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c" exitCode=0 Jul 16 12:12:05.893486 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.893407 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" Jul 16 12:12:05.893531 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.893408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" event={"ID":"388a0ec4-564c-4a5f-8b4d-487e8ef9742b","Type":"ContainerDied","Data":"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c"} Jul 16 12:12:05.893575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.893530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5854b8c858-7t9j4" event={"ID":"388a0ec4-564c-4a5f-8b4d-487e8ef9742b","Type":"ContainerDied","Data":"0bd82559d957592f1b3c1a32176a7dde9bb0b0f52996978e51cc95fc35c328d2"} Jul 16 12:12:05.893575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.893559 2583 scope.go:117] "RemoveContainer" containerID="c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c" Jul 16 12:12:05.901961 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.901943 2583 scope.go:117] "RemoveContainer" containerID="c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c" Jul 16 12:12:05.902244 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:12:05.902225 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c\": container with ID starting with c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c not found: ID does not exist" containerID="c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c" Jul 16 12:12:05.902301 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.902252 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c"} err="failed to get container status \"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c\": rpc error: code = NotFound desc = could not find container \"c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c\": container with ID starting with c48b756a3bce62c4725850ef8f64cf81e2852e329a2d2335cf42f58bebeef11c not found: ID does not exist" Jul 16 12:12:05.925457 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.925434 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:12:05.931348 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:05.931327 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5854b8c858-7t9j4"] Jul 16 12:12:06.792668 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:06.792634 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" path="/var/lib/kubelet/pods/388a0ec4-564c-4a5f-8b4d-487e8ef9742b/volumes" Jul 16 12:12:10.082007 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:10.081976 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggfxd_e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3/dns/0.log" Jul 16 12:12:10.274879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:10.274847 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggfxd_e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3/kube-rbac-proxy/0.log" Jul 16 12:12:11.083491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:11.083460 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbpk4_1d320534-8242-4791-8292-7fbfca21d331/dns-node-resolver/0.log" Jul 16 12:12:11.867713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:12:11.867682 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qt9pz_de45197d-cd2f-4624-b972-cc76efc7ed86/serve-healthcheck-canary/0.log" Jul 16 12:14:28.431228 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.431189 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf"] Jul 16 12:14:28.431701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.431420 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" containerName="registry" Jul 16 12:14:28.431701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.431431 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" containerName="registry" Jul 16 12:14:28.431701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.431474 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="388a0ec4-564c-4a5f-8b4d-487e8ef9742b" containerName="registry" Jul 16 12:14:28.434188 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.434171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.438505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.438476 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 12:14:28.438505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.438487 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 12:14:28.438673 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.438542 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jk6rl\"" Jul 16 12:14:28.456311 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.456280 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf"] Jul 16 12:14:28.603821 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.603778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvb2b\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-kube-api-access-kvb2b\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.603821 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.603822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.704595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.704484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvb2b\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-kube-api-access-kvb2b\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.704595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.704547 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.714857 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.714822 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.715336 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.715315 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvb2b\" (UniqueName: \"kubernetes.io/projected/7aa2b9d6-e141-4144-a86d-bf1f27cbe557-kube-api-access-kvb2b\") pod \"cert-manager-cainjector-559cdd8c87-n5wsf\" (UID: \"7aa2b9d6-e141-4144-a86d-bf1f27cbe557\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.742702 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.742664 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" Jul 16 12:14:28.881421 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.881391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf"] Jul 16 12:14:28.883695 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:14:28.883666 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aa2b9d6_e141_4144_a86d_bf1f27cbe557.slice/crio-0f6075deeff95f9a730ebea35b4c86a46ec3b2feac59e904b6913d37b3277b8f WatchSource:0}: Error finding container 0f6075deeff95f9a730ebea35b4c86a46ec3b2feac59e904b6913d37b3277b8f: Status 404 returned error can't find the container with id 0f6075deeff95f9a730ebea35b4c86a46ec3b2feac59e904b6913d37b3277b8f Jul 16 12:14:28.885664 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:28.885647 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:14:29.359564 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:29.359524 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" event={"ID":"7aa2b9d6-e141-4144-a86d-bf1f27cbe557","Type":"ContainerStarted","Data":"0f6075deeff95f9a730ebea35b4c86a46ec3b2feac59e904b6913d37b3277b8f"} Jul 16 12:14:32.375863 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:32.375829 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" event={"ID":"7aa2b9d6-e141-4144-a86d-bf1f27cbe557","Type":"ContainerStarted","Data":"f9862bdeae601b95d2bb40bbce9c5e92a4028ab03aa30653e7f20ddb452a3644"} Jul 16 12:14:32.402313 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:14:32.402255 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n5wsf" podStartSLOduration=1.363126176 podStartE2EDuration="4.402235041s" totalImagesPullingTime="3.039108865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:14:32.401487617 +0000 UTC m=+470.095594338" watchObservedRunningTime="2026-07-16 12:14:32.402235041 +0000 UTC m=+470.096341755" Jul 16 12:15:01.316735 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.316702 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx"] Jul 16 12:15:01.319754 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.319738 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.323714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.323671 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 12:15:01.323844 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.323830 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 12:15:01.324193 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.324174 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 12:15:01.324193 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.324186 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9js4\"" Jul 16 12:15:01.324355 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.324174 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 12:15:01.363126 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.363102 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx"] Jul 16 12:15:01.420976 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.420945 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.420976 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.420977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.421150 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.421050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.421150 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.421080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.421150 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.421123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522081 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522053 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522236 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522236 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522124 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522236 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522236 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.522462 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.522444 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-manifests\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.524369 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.524345 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-tmp\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.524605 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.524587 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.524706 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.524682 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.534558 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.534533 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsz5\" (UniqueName: \"kubernetes.io/projected/e641c5e4-bc91-4b0e-96b8-3b0f9265150f-kube-api-access-8vsz5\") pod \"opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx\" (UID: \"e641c5e4-bc91-4b0e-96b8-3b0f9265150f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.629829 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.629794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:01.786795 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:01.786767 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx"] Jul 16 12:15:01.789584 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:01.789559 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode641c5e4_bc91_4b0e_96b8_3b0f9265150f.slice/crio-440a83dc42591a2f118d3a86c1932c5aec07aa2be92f65ed94bf1ea2315f4bb0 WatchSource:0}: Error finding container 440a83dc42591a2f118d3a86c1932c5aec07aa2be92f65ed94bf1ea2315f4bb0: Status 404 returned error can't find the container with id 440a83dc42591a2f118d3a86c1932c5aec07aa2be92f65ed94bf1ea2315f4bb0 Jul 16 12:15:02.478947 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:02.478912 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" event={"ID":"e641c5e4-bc91-4b0e-96b8-3b0f9265150f","Type":"ContainerStarted","Data":"440a83dc42591a2f118d3a86c1932c5aec07aa2be92f65ed94bf1ea2315f4bb0"} Jul 16 12:15:05.491581 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:05.491545 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" event={"ID":"e641c5e4-bc91-4b0e-96b8-3b0f9265150f","Type":"ContainerStarted","Data":"d1eb93331a29c6da1f3e534bfcace367a6b23a53444c64bbe6767840759de8ca"} Jul 16 12:15:06.495847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:06.495816 2583 generic.go:356] "Generic (PLEG): container finished" podID="e641c5e4-bc91-4b0e-96b8-3b0f9265150f" containerID="d1eb93331a29c6da1f3e534bfcace367a6b23a53444c64bbe6767840759de8ca" exitCode=0 Jul 16 12:15:06.496242 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:06.495858 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" event={"ID":"e641c5e4-bc91-4b0e-96b8-3b0f9265150f","Type":"ContainerDied","Data":"d1eb93331a29c6da1f3e534bfcace367a6b23a53444c64bbe6767840759de8ca"} Jul 16 12:15:07.501480 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:07.501444 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" event={"ID":"e641c5e4-bc91-4b0e-96b8-3b0f9265150f","Type":"ContainerStarted","Data":"4ab753f5c671bbd0dac9808e976cd98a919fb08f87efa325e3681e2f83fbcfc6"} Jul 16 12:15:07.501848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:07.501566 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:18.507341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:18.507311 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" Jul 16 12:15:18.538728 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:18.538673 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx" podStartSLOduration=12.441964071 podStartE2EDuration="17.538658629s" totalImagesPullingTime="4.111271633s" totalInitContainerRuntime="985.422925ms" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:07.561905761 +0000 UTC m=+505.256012480" watchObservedRunningTime="2026-07-16 12:15:18.538658629 +0000 UTC m=+516.232765347" Jul 16 12:15:24.085351 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.085310 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4cqq4"] Jul 16 12:15:24.093521 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.093491 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.102568 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.102539 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 12:15:24.102734 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.102568 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qqx27\"" Jul 16 12:15:24.114984 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.114947 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4cqq4"] Jul 16 12:15:24.193127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.193083 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsb5c\" (UniqueName: \"kubernetes.io/projected/51a5f9f9-eb1f-48a9-b914-622742efb141-kube-api-access-wsb5c\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.193127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.193133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51a5f9f9-eb1f-48a9-b914-622742efb141-cert\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.294638 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.294596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsb5c\" (UniqueName: \"kubernetes.io/projected/51a5f9f9-eb1f-48a9-b914-622742efb141-kube-api-access-wsb5c\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.294638 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.294636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51a5f9f9-eb1f-48a9-b914-622742efb141-cert\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.297173 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.297143 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51a5f9f9-eb1f-48a9-b914-622742efb141-cert\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.316317 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.316285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsb5c\" (UniqueName: \"kubernetes.io/projected/51a5f9f9-eb1f-48a9-b914-622742efb141-kube-api-access-wsb5c\") pod \"odh-model-controller-7998c67dc4-4cqq4\" (UID: \"51a5f9f9-eb1f-48a9-b914-622742efb141\") " pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.403931 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.403889 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:24.560422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:24.560390 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4cqq4"] Jul 16 12:15:24.562429 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:24.562384 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a5f9f9_eb1f_48a9_b914_622742efb141.slice/crio-179c23b7c7d019bc739b0634f95dfa2d5bdc182930f5740e275c4b0ec473f409 WatchSource:0}: Error finding container 179c23b7c7d019bc739b0634f95dfa2d5bdc182930f5740e275c4b0ec473f409: Status 404 returned error can't find the container with id 179c23b7c7d019bc739b0634f95dfa2d5bdc182930f5740e275c4b0ec473f409 Jul 16 12:15:25.241515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.241479 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-52ws8"] Jul 16 12:15:25.245006 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.244958 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.255193 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.255158 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 16 12:15:25.260103 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.259958 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-52ws8"] Jul 16 12:15:25.303129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.302935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.303129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.303024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xjz\" (UniqueName: \"kubernetes.io/projected/684cf32e-3332-4065-973e-70aa82907ae4-kube-api-access-l5xjz\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.404098 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.404056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.404313 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.404113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xjz\" (UniqueName: \"kubernetes.io/projected/684cf32e-3332-4065-973e-70aa82907ae4-kube-api-access-l5xjz\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.404313 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:15:25.404243 2583 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 16 12:15:25.404433 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:15:25.404326 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs podName:684cf32e-3332-4065-973e-70aa82907ae4 nodeName:}" failed. No retries permitted until 2026-07-16 12:15:25.904299409 +0000 UTC m=+523.598406125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs") pod "model-serving-api-54d89bd5fc-52ws8" (UID: "684cf32e-3332-4065-973e-70aa82907ae4") : secret "model-serving-api-tls" not found Jul 16 12:15:25.420109 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.420037 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xjz\" (UniqueName: \"kubernetes.io/projected/684cf32e-3332-4065-973e-70aa82907ae4-kube-api-access-l5xjz\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.570143 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.570035 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" event={"ID":"51a5f9f9-eb1f-48a9-b914-622742efb141","Type":"ContainerStarted","Data":"179c23b7c7d019bc739b0634f95dfa2d5bdc182930f5740e275c4b0ec473f409"} Jul 16 12:15:25.909316 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.909247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:25.912506 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:25.912445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684cf32e-3332-4065-973e-70aa82907ae4-tls-certs\") pod \"model-serving-api-54d89bd5fc-52ws8\" (UID: \"684cf32e-3332-4065-973e-70aa82907ae4\") " pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:26.161448 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:26.161360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:27.460602 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:27.460557 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-52ws8"] Jul 16 12:15:27.464876 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:27.464835 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684cf32e_3332_4065_973e_70aa82907ae4.slice/crio-8a6a2119a1ea451f79a1dc41d3f4b9b9bc3e9ba220e337c99b47ff5c7eb46146 WatchSource:0}: Error finding container 8a6a2119a1ea451f79a1dc41d3f4b9b9bc3e9ba220e337c99b47ff5c7eb46146: Status 404 returned error can't find the container with id 8a6a2119a1ea451f79a1dc41d3f4b9b9bc3e9ba220e337c99b47ff5c7eb46146 Jul 16 12:15:27.580569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:27.580456 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" event={"ID":"684cf32e-3332-4065-973e-70aa82907ae4","Type":"ContainerStarted","Data":"8a6a2119a1ea451f79a1dc41d3f4b9b9bc3e9ba220e337c99b47ff5c7eb46146"} Jul 16 12:15:27.582153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:27.582118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" event={"ID":"51a5f9f9-eb1f-48a9-b914-622742efb141","Type":"ContainerStarted","Data":"f816c85a2fe961067f770bd65d91570bdfb3733b6f429c021ec8e0208e6cf850"} Jul 16 12:15:27.582259 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:27.582240 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:27.615274 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:27.615215 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" podStartSLOduration=0.811333256 podStartE2EDuration="3.615196005s" totalImagesPullingTime="2.803862749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:27.60973482 +0000 UTC m=+525.303841538" watchObservedRunningTime="2026-07-16 12:15:27.615196005 +0000 UTC m=+525.309302724" Jul 16 12:15:29.596546 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.596488 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" event={"ID":"684cf32e-3332-4065-973e-70aa82907ae4","Type":"ContainerStarted","Data":"8e2bdce5ad7b73c3687f0b3221e3ba4eef5e212361090957eade27224cc3a973"} Jul 16 12:15:29.597032 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.596606 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:29.633325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.633259 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" podStartSLOduration=2.696297004 podStartE2EDuration="4.633234907s" totalImagesPullingTime="1.936937903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:29.631894538 +0000 UTC m=+527.326001256" watchObservedRunningTime="2026-07-16 12:15:29.633234907 +0000 UTC m=+527.327341628" Jul 16 12:15:29.716614 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.716511 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq"] Jul 16 12:15:29.719918 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.719884 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.723264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.723225 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 12:15:29.723264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.723219 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 12:15:29.723264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.723221 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 12:15:29.723530 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.723220 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 12:15:29.742157 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.742119 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq"] Jul 16 12:15:29.843077 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.843024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742ae0a-cc41-445f-b477-526b52f9093e-tls-certs\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.843302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.843146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85pq\" (UniqueName: \"kubernetes.io/projected/4742ae0a-cc41-445f-b477-526b52f9093e-kube-api-access-f85pq\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.843302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.843219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4742ae0a-cc41-445f-b477-526b52f9093e-tmp\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.944309 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.944265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742ae0a-cc41-445f-b477-526b52f9093e-tls-certs\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.944540 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.944326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85pq\" (UniqueName: \"kubernetes.io/projected/4742ae0a-cc41-445f-b477-526b52f9093e-kube-api-access-f85pq\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.944540 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.944372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4742ae0a-cc41-445f-b477-526b52f9093e-tmp\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.947055 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.947019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4742ae0a-cc41-445f-b477-526b52f9093e-tmp\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.947202 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.947188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742ae0a-cc41-445f-b477-526b52f9093e-tls-certs\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:29.957354 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:29.957325 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85pq\" (UniqueName: \"kubernetes.io/projected/4742ae0a-cc41-445f-b477-526b52f9093e-kube-api-access-f85pq\") pod \"kube-auth-proxy-569f4f6548-wzwpq\" (UID: \"4742ae0a-cc41-445f-b477-526b52f9093e\") " pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:30.032085 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:30.031972 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" Jul 16 12:15:30.196416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:30.196379 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq"] Jul 16 12:15:30.199762 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:30.199721 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4742ae0a_cc41_445f_b477_526b52f9093e.slice/crio-00b26075deb84e7888025e50f995c38f186a1e1bc02b52c02a4a62dee9b049a8 WatchSource:0}: Error finding container 00b26075deb84e7888025e50f995c38f186a1e1bc02b52c02a4a62dee9b049a8: Status 404 returned error can't find the container with id 00b26075deb84e7888025e50f995c38f186a1e1bc02b52c02a4a62dee9b049a8 Jul 16 12:15:30.603205 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:30.603165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" event={"ID":"4742ae0a-cc41-445f-b477-526b52f9093e","Type":"ContainerStarted","Data":"00b26075deb84e7888025e50f995c38f186a1e1bc02b52c02a4a62dee9b049a8"} Jul 16 12:15:31.850270 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.850231 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:31.854556 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.854533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.893240 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.893202 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 12:15:31.893458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.893262 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 12:15:31.893458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.893269 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 12:15:31.893458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.893391 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 12:15:31.893458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.893407 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j6lrv\"" Jul 16 12:15:31.898050 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.898019 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:31.962641 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.962641 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962642 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.962889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.962889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.962889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.962889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:31.963061 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:31.962914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6p9h\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063343 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063703 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063703 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6p9h\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.063703 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.063672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.064328 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.064296 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.066140 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.066105 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.066253 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.066191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.066357 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.066341 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.066467 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.066446 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.075169 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.075141 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6p9h\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.080262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.080229 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8d6mz\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.164470 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.164434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:32.355225 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:32.355193 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9b68b6_9c38_47c0_a564_2c1e522709e2.slice/crio-4738ef03a8c2f0f9788a561d2d3a6dd65c6b23de9134bec35eb9a0ac818ed896 WatchSource:0}: Error finding container 4738ef03a8c2f0f9788a561d2d3a6dd65c6b23de9134bec35eb9a0ac818ed896: Status 404 returned error can't find the container with id 4738ef03a8c2f0f9788a561d2d3a6dd65c6b23de9134bec35eb9a0ac818ed896 Jul 16 12:15:32.355407 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.355382 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:32.614780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:32.614686 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" event={"ID":"9f9b68b6-9c38-47c0-a564-2c1e522709e2","Type":"ContainerStarted","Data":"4738ef03a8c2f0f9788a561d2d3a6dd65c6b23de9134bec35eb9a0ac818ed896"} Jul 16 12:15:35.897775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:35.897715 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:15:35.898221 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:35.897823 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:15:36.482809 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.482772 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5czs6"] Jul 16 12:15:36.485906 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.485885 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.493740 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.493715 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f7w72\"" Jul 16 12:15:36.494021 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.493980 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 16 12:15:36.505847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.505810 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw"] Jul 16 12:15:36.509183 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.509154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.512029 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.512002 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tfznq\"" Jul 16 12:15:36.512772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.512755 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 16 12:15:36.546491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.546453 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5czs6"] Jul 16 12:15:36.567027 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.566966 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw"] Jul 16 12:15:36.602849 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.602806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.602849 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.602849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5kn\" (UniqueName: \"kubernetes.io/projected/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-kube-api-access-rc5kn\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.603137 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.602877 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fba93b3-1602-4b38-901b-ee0facca0567-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.603137 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.603016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scr4\" (UniqueName: \"kubernetes.io/projected/7fba93b3-1602-4b38-901b-ee0facca0567-kube-api-access-7scr4\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.638223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.638177 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" event={"ID":"9f9b68b6-9c38-47c0-a564-2c1e522709e2","Type":"ContainerStarted","Data":"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc"} Jul 16 12:15:36.638404 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.638244 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:36.640743 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.640703 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" event={"ID":"4742ae0a-cc41-445f-b477-526b52f9093e","Type":"ContainerStarted","Data":"8280ba183d28b296947071620ec4551c4329cf826c52ec40076010d136c5a0cf"} Jul 16 12:15:36.640923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.640796 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-8d6mz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 12:15:36.640923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.640893 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 12:15:36.681442 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.681375 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" podStartSLOduration=2.141506076 podStartE2EDuration="5.681352168s" totalImagesPullingTime="3.539846092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:36.679485176 +0000 UTC m=+534.373591944" watchObservedRunningTime="2026-07-16 12:15:36.681352168 +0000 UTC m=+534.375458888" Jul 16 12:15:36.704571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.703867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fba93b3-1602-4b38-901b-ee0facca0567-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.704571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.703961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scr4\" (UniqueName: \"kubernetes.io/projected/7fba93b3-1602-4b38-901b-ee0facca0567-kube-api-access-7scr4\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.704571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.704027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.704571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.704055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5kn\" (UniqueName: \"kubernetes.io/projected/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-kube-api-access-rc5kn\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.704926 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:15:36.704769 2583 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 16 12:15:36.704926 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:15:36.704860 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert podName:3ca17d96-66a9-4bcf-8783-e7cf20ca6545 nodeName:}" failed. No retries permitted until 2026-07-16 12:15:37.20483842 +0000 UTC m=+534.898945127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert") pod "kserve-controller-manager-d6f4b65b8-5czs6" (UID: "3ca17d96-66a9-4bcf-8783-e7cf20ca6545") : secret "kserve-webhook-server-cert" not found Jul 16 12:15:36.707447 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.707415 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fba93b3-1602-4b38-901b-ee0facca0567-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.720717 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.720670 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5kn\" (UniqueName: \"kubernetes.io/projected/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-kube-api-access-rc5kn\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:36.724651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.724614 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scr4\" (UniqueName: \"kubernetes.io/projected/7fba93b3-1602-4b38-901b-ee0facca0567-kube-api-access-7scr4\") pod \"llmisvc-controller-manager-777c6bd7d5-bw7dw\" (UID: \"7fba93b3-1602-4b38-901b-ee0facca0567\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:36.727580 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.727497 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-569f4f6548-wzwpq" podStartSLOduration=2.029029503 podStartE2EDuration="7.72747685s" totalImagesPullingTime="5.698447347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:36.726406119 +0000 UTC m=+534.420512840" watchObservedRunningTime="2026-07-16 12:15:36.72747685 +0000 UTC m=+534.421583570" Jul 16 12:15:36.820381 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:36.820284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:37.065779 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.065741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw"] Jul 16 12:15:37.077611 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:37.077576 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7fba93b3_1602_4b38_901b_ee0facca0567.slice/crio-bdfae32c67a6b1acc7da00a24d9641415a604e71c6272ba05777df350d699271 WatchSource:0}: Error finding container bdfae32c67a6b1acc7da00a24d9641415a604e71c6272ba05777df350d699271: Status 404 returned error can't find the container with id bdfae32c67a6b1acc7da00a24d9641415a604e71c6272ba05777df350d699271 Jul 16 12:15:37.209509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.209459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:37.212417 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.212373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca17d96-66a9-4bcf-8783-e7cf20ca6545-cert\") pod \"kserve-controller-manager-d6f4b65b8-5czs6\" (UID: \"3ca17d96-66a9-4bcf-8783-e7cf20ca6545\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:37.396930 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.396887 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:37.602714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.602678 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5czs6"] Jul 16 12:15:37.606050 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:15:37.606016 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca17d96_66a9_4bcf_8783_e7cf20ca6545.slice/crio-bc051f08c69be4e3e8a5407cfbaf7dcbfeb809a811d73258af3dea7d0432c95f WatchSource:0}: Error finding container bc051f08c69be4e3e8a5407cfbaf7dcbfeb809a811d73258af3dea7d0432c95f: Status 404 returned error can't find the container with id bc051f08c69be4e3e8a5407cfbaf7dcbfeb809a811d73258af3dea7d0432c95f Jul 16 12:15:37.650313 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.648372 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" event={"ID":"7fba93b3-1602-4b38-901b-ee0facca0567","Type":"ContainerStarted","Data":"bdfae32c67a6b1acc7da00a24d9641415a604e71c6272ba05777df350d699271"} Jul 16 12:15:37.651121 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.651082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" event={"ID":"3ca17d96-66a9-4bcf-8783-e7cf20ca6545","Type":"ContainerStarted","Data":"bc051f08c69be4e3e8a5407cfbaf7dcbfeb809a811d73258af3dea7d0432c95f"} Jul 16 12:15:37.651903 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.651876 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-8d6mz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 12:15:37.652026 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:37.651927 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 12:15:38.588708 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:38.588665 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4cqq4" Jul 16 12:15:40.102563 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.102524 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j"] Jul 16 12:15:40.107083 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.107058 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.111727 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.111701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4q8hb\"" Jul 16 12:15:40.140401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.140370 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j"] Jul 16 12:15:40.239615 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239574 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xv5\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-kube-api-access-s5xv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239735 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239762 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239943 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239943 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239833 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239943 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.239943 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.239906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d701f5ac-2bbf-4073-9668-e22bface55b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340435 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340629 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d701f5ac-2bbf-4073-9668-e22bface55b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340887 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340887 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xv5\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-kube-api-access-s5xv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340887 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.340887 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340879 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.341139 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.340972 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.341139 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.341068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.341468 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.341442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.341750 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.341706 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d701f5ac-2bbf-4073-9668-e22bface55b5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.343398 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.343370 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.343517 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.343398 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.376907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.376873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.386665 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.386640 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xv5\" (UniqueName: \"kubernetes.io/projected/d701f5ac-2bbf-4073-9668-e22bface55b5-kube-api-access-s5xv5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49pz5j\" (UID: \"d701f5ac-2bbf-4073-9668-e22bface55b5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.420597 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.420516 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:40.608046 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.608017 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-52ws8" Jul 16 12:15:40.646138 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.646051 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:40.646305 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.646256 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" containerID="cri-o://2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc" gracePeriod=30 Jul 16 12:15:40.646924 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:40.646893 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:41.267889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.267866 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:41.348672 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.348634 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j"] Jul 16 12:15:41.349852 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.349535 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.349852 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.349578 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.349852 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.349608 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.349852 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.349633 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.350078 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.349963 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6p9h\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.350197 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.350087 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.350252 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.350209 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert\") pod \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\" (UID: \"9f9b68b6-9c38-47c0-a564-2c1e522709e2\") " Jul 16 12:15:41.350793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.350754 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:15:41.355709 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.354231 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h" (OuterVolumeSpecName: "kube-api-access-r6p9h") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "kube-api-access-r6p9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:15:41.358662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.356646 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:15:41.358662 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.357199 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:15:41.359170 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.359061 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts" (OuterVolumeSpecName: "cacerts") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:15:41.359234 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.359184 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:15:41.359466 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.359439 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs" (OuterVolumeSpecName: "local-certs") pod "9f9b68b6-9c38-47c0-a564-2c1e522709e2" (UID: "9f9b68b6-9c38-47c0-a564-2c1e522709e2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:15:41.451604 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451578 2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-dns-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451604 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451602 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451612 2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-cacerts\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451620 2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f9b68b6-9c38-47c0-a564-2c1e522709e2-local-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451627 2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-kubeconfig\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451635 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6p9h\" (UniqueName: \"kubernetes.io/projected/9f9b68b6-9c38-47c0-a564-2c1e522709e2-kube-api-access-r6p9h\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.451770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.451644 2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f9b68b6-9c38-47c0-a564-2c1e522709e2-istio-csr-ca-configmap\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:15:41.670452 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.670397 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" event={"ID":"7fba93b3-1602-4b38-901b-ee0facca0567","Type":"ContainerStarted","Data":"edc20079b170cac94e139f0a4e7daeedd813bf6a505dc3b817ceb77f32b346f9"} Jul 16 12:15:41.670621 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.670565 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:15:41.671660 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.671607 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" event={"ID":"d701f5ac-2bbf-4073-9668-e22bface55b5","Type":"ContainerStarted","Data":"695fc55798458a23b06db7986c3cf51806b27888d969ee34229962222e45d826"} Jul 16 12:15:41.672877 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.672856 2583 generic.go:356] "Generic (PLEG): container finished" podID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerID="2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc" exitCode=0 Jul 16 12:15:41.673008 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.672920 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" event={"ID":"9f9b68b6-9c38-47c0-a564-2c1e522709e2","Type":"ContainerDied","Data":"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc"} Jul 16 12:15:41.673008 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.672921 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" Jul 16 12:15:41.673008 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.672951 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz" event={"ID":"9f9b68b6-9c38-47c0-a564-2c1e522709e2","Type":"ContainerDied","Data":"4738ef03a8c2f0f9788a561d2d3a6dd65c6b23de9134bec35eb9a0ac818ed896"} Jul 16 12:15:41.673008 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.672973 2583 scope.go:117] "RemoveContainer" containerID="2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc" Jul 16 12:15:41.674638 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.674611 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" event={"ID":"3ca17d96-66a9-4bcf-8783-e7cf20ca6545","Type":"ContainerStarted","Data":"304c209966d8d81fad8cb52c1a9e65a6c31791ca1703136587db731dd640c2e1"} Jul 16 12:15:41.674780 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.674766 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:15:41.682063 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.682045 2583 scope.go:117] "RemoveContainer" containerID="2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc" Jul 16 12:15:41.682308 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:15:41.682288 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc\": container with ID starting with 2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc not found: ID does not exist" containerID="2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc" Jul 16 12:15:41.682402 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.682316 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc"} err="failed to get container status \"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc\": rpc error: code = NotFound desc = could not find container \"2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc\": container with ID starting with 2fd2bc5fc7204e8d5d7a6bc620161fe6a20bd6e42cb0c8e5737248285ceacfbc not found: ID does not exist" Jul 16 12:15:41.756169 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.756079 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" podStartSLOduration=1.6815524549999998 podStartE2EDuration="5.756065292s" totalImagesPullingTime="4.074512837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:41.755306539 +0000 UTC m=+539.449413269" watchObservedRunningTime="2026-07-16 12:15:41.756065292 +0000 UTC m=+539.450172011" Jul 16 12:15:41.794094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.794070 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:41.797528 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.797506 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8d6mz"] Jul 16 12:15:41.826092 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:41.826052 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" podStartSLOduration=2.282097909 podStartE2EDuration="5.826039497s" totalImagesPullingTime="3.543941588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:41.823813718 +0000 UTC m=+539.517920437" watchObservedRunningTime="2026-07-16 12:15:41.826039497 +0000 UTC m=+539.520146215" Jul 16 12:15:42.794517 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:42.794485 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" path="/var/lib/kubelet/pods/9f9b68b6-9c38-47c0-a564-2c1e522709e2/volumes" Jul 16 12:15:44.264687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:44.264645 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:15:44.264948 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:44.264721 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:15:44.264948 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:44.264750 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:15:44.690308 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:44.690272 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" event={"ID":"d701f5ac-2bbf-4073-9668-e22bface55b5","Type":"ContainerStarted","Data":"dd531c963094fea67931f80ad362bb6ee1a332e9c587e67eef3c485895e36393"} Jul 16 12:15:44.726676 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:44.726608 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" podStartSLOduration=1.819734787 podStartE2EDuration="4.726588331s" totalImagesPullingTime="2.906853544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:15:44.725161169 +0000 UTC m=+542.419267906" watchObservedRunningTime="2026-07-16 12:15:44.726588331 +0000 UTC m=+542.420695053" Jul 16 12:15:45.420966 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:45.420929 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:45.425097 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:45.425072 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:45.694450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:45.694369 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:15:45.695226 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:15:45.695208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49pz5j" Jul 16 12:16:12.683490 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:12.683459 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bw7dw" Jul 16 12:16:12.686575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:12.686553 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5czs6" Jul 16 12:16:24.807539 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.807506 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:24.807897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.807793 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" Jul 16 12:16:24.807897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.807809 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" Jul 16 12:16:24.807897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.807862 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f9b68b6-9c38-47c0-a564-2c1e522709e2" containerName="discovery" Jul 16 12:16:24.815165 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.815143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:24.820307 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.820280 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-ssbmn\"" Jul 16 12:16:24.820447 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.820392 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:16:24.821338 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.821316 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:16:24.833836 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.833816 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:24.864303 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.864275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmb4\" (UniqueName: \"kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4\") pod \"kuadrant-operator-catalog-xvsrf\" (UID: \"93f64ec0-ee39-401c-9b6e-99419667ad2a\") " pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:24.965744 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.965709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmb4\" (UniqueName: \"kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4\") pod \"kuadrant-operator-catalog-xvsrf\" (UID: \"93f64ec0-ee39-401c-9b6e-99419667ad2a\") " pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:24.977879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:24.977850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmb4\" (UniqueName: \"kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4\") pod \"kuadrant-operator-catalog-xvsrf\" (UID: \"93f64ec0-ee39-401c-9b6e-99419667ad2a\") " pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:25.125076 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.125046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:25.146790 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.146767 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:25.276578 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.276546 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:25.280332 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:16:25.280299 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f64ec0_ee39_401c_9b6e_99419667ad2a.slice/crio-ec0a16834702363966a1a63da0f1fe5be50937b0f948032d3d0e840a63095afc WatchSource:0}: Error finding container ec0a16834702363966a1a63da0f1fe5be50937b0f948032d3d0e840a63095afc: Status 404 returned error can't find the container with id ec0a16834702363966a1a63da0f1fe5be50937b0f948032d3d0e840a63095afc Jul 16 12:16:25.381143 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.381057 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z5rbn"] Jul 16 12:16:25.384045 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.384029 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:25.398325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.398292 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z5rbn"] Jul 16 12:16:25.471793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.471759 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6fg\" (UniqueName: \"kubernetes.io/projected/8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c-kube-api-access-md6fg\") pod \"kuadrant-operator-catalog-z5rbn\" (UID: \"8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c\") " pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:25.572466 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.572431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6fg\" (UniqueName: \"kubernetes.io/projected/8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c-kube-api-access-md6fg\") pod \"kuadrant-operator-catalog-z5rbn\" (UID: \"8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c\") " pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:25.591307 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.591269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6fg\" (UniqueName: \"kubernetes.io/projected/8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c-kube-api-access-md6fg\") pod \"kuadrant-operator-catalog-z5rbn\" (UID: \"8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c\") " pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:25.693094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.693010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:25.843140 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.843112 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z5rbn"] Jul 16 12:16:25.856437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:25.856405 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" event={"ID":"93f64ec0-ee39-401c-9b6e-99419667ad2a","Type":"ContainerStarted","Data":"ec0a16834702363966a1a63da0f1fe5be50937b0f948032d3d0e840a63095afc"} Jul 16 12:16:25.860408 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:16:25.860381 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2eaa1b_1d25_40eb_bb3f_fd0c9912759c.slice/crio-1aaea6f27c6649d6d87ce54f6ad0f842e007fc516635f5bdcdf826ae9bdfb142 WatchSource:0}: Error finding container 1aaea6f27c6649d6d87ce54f6ad0f842e007fc516635f5bdcdf826ae9bdfb142: Status 404 returned error can't find the container with id 1aaea6f27c6649d6d87ce54f6ad0f842e007fc516635f5bdcdf826ae9bdfb142 Jul 16 12:16:26.862812 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:26.862757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" event={"ID":"8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c","Type":"ContainerStarted","Data":"1aaea6f27c6649d6d87ce54f6ad0f842e007fc516635f5bdcdf826ae9bdfb142"} Jul 16 12:16:27.867848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:27.867808 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" event={"ID":"93f64ec0-ee39-401c-9b6e-99419667ad2a","Type":"ContainerStarted","Data":"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457"} Jul 16 12:16:27.868328 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:27.867854 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" podUID="93f64ec0-ee39-401c-9b6e-99419667ad2a" containerName="registry-server" containerID="cri-o://8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457" gracePeriod=2 Jul 16 12:16:27.869281 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:27.869257 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" event={"ID":"8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c","Type":"ContainerStarted","Data":"a5a3fd29275c5a61b7c4f43873a86e88e1e03491bd857c23af32b59aa6418051"} Jul 16 12:16:27.918767 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:27.918706 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" podStartSLOduration=1.993059996 podStartE2EDuration="3.918684354s" totalImagesPullingTime="1.925624358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:16:27.890862777 +0000 UTC m=+585.584969515" watchObservedRunningTime="2026-07-16 12:16:27.918684354 +0000 UTC m=+585.612791074" Jul 16 12:16:27.919906 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:27.919864 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" podStartSLOduration=1.573703968 podStartE2EDuration="2.919851236s" totalImagesPullingTime="1.346147268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:16:27.918394939 +0000 UTC m=+585.612501685" watchObservedRunningTime="2026-07-16 12:16:27.919851236 +0000 UTC m=+585.613957955" Jul 16 12:16:28.107559 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.107533 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:28.193341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.193308 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvmb4\" (UniqueName: \"kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4\") pod \"93f64ec0-ee39-401c-9b6e-99419667ad2a\" (UID: \"93f64ec0-ee39-401c-9b6e-99419667ad2a\") " Jul 16 12:16:28.195427 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.195400 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4" (OuterVolumeSpecName: "kube-api-access-zvmb4") pod "93f64ec0-ee39-401c-9b6e-99419667ad2a" (UID: "93f64ec0-ee39-401c-9b6e-99419667ad2a"). InnerVolumeSpecName "kube-api-access-zvmb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:16:28.294224 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.294192 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmb4\" (UniqueName: \"kubernetes.io/projected/93f64ec0-ee39-401c-9b6e-99419667ad2a-kube-api-access-zvmb4\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:16:28.874894 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.874858 2583 generic.go:356] "Generic (PLEG): container finished" podID="93f64ec0-ee39-401c-9b6e-99419667ad2a" containerID="8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457" exitCode=0 Jul 16 12:16:28.875333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.874902 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" Jul 16 12:16:28.875333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.874946 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" event={"ID":"93f64ec0-ee39-401c-9b6e-99419667ad2a","Type":"ContainerDied","Data":"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457"} Jul 16 12:16:28.875333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.875016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xvsrf" event={"ID":"93f64ec0-ee39-401c-9b6e-99419667ad2a","Type":"ContainerDied","Data":"ec0a16834702363966a1a63da0f1fe5be50937b0f948032d3d0e840a63095afc"} Jul 16 12:16:28.875333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.875047 2583 scope.go:117] "RemoveContainer" containerID="8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457" Jul 16 12:16:28.883718 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.883700 2583 scope.go:117] "RemoveContainer" containerID="8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457" Jul 16 12:16:28.883984 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:16:28.883966 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457\": container with ID starting with 8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457 not found: ID does not exist" containerID="8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457" Jul 16 12:16:28.884069 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.884018 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457"} err="failed to get container status \"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457\": rpc error: code = NotFound desc = could not find container \"8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457\": container with ID starting with 8e4dac360e137cd1c0764c47c63d28059ad7f9e9672de3040726c68358796457 not found: ID does not exist" Jul 16 12:16:28.898857 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.898827 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:28.904778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:28.904754 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xvsrf"] Jul 16 12:16:30.793612 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:30.793579 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f64ec0-ee39-401c-9b6e-99419667ad2a" path="/var/lib/kubelet/pods/93f64ec0-ee39-401c-9b6e-99419667ad2a/volumes" Jul 16 12:16:35.693175 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:35.693130 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:35.693600 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:35.693314 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:35.716091 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:35.716061 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:35.925023 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:35.924973 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-z5rbn" Jul 16 12:16:42.757523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:42.757486 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:16:42.757947 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:42.757578 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:16:59.007778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.007724 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt"] Jul 16 12:16:59.008209 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.008041 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f64ec0-ee39-401c-9b6e-99419667ad2a" containerName="registry-server" Jul 16 12:16:59.008209 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.008054 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f64ec0-ee39-401c-9b6e-99419667ad2a" containerName="registry-server" Jul 16 12:16:59.008209 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.008122 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f64ec0-ee39-401c-9b6e-99419667ad2a" containerName="registry-server" Jul 16 12:16:59.011794 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.011776 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:16:59.015153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.015136 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bs4zq\"" Jul 16 12:16:59.024556 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.024533 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt"] Jul 16 12:16:59.141759 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.141720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhr4\" (UniqueName: \"kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4\") pod \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" (UID: \"78449c1d-cfda-43fd-986f-240ef71706ec\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:16:59.242358 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.242320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhr4\" (UniqueName: \"kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4\") pod \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" (UID: \"78449c1d-cfda-43fd-986f-240ef71706ec\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:16:59.257635 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.257598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhr4\" (UniqueName: \"kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4\") pod \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" (UID: \"78449c1d-cfda-43fd-986f-240ef71706ec\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:16:59.321668 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.321575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:16:59.463701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:16:59.463664 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt"] Jul 16 12:16:59.466969 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:16:59.466941 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78449c1d_cfda_43fd_986f_240ef71706ec.slice/crio-ea67f9d30541ec18b674180717367f12a638d6a1ec53a8c1dbe2c49a4e3f64cb WatchSource:0}: Error finding container ea67f9d30541ec18b674180717367f12a638d6a1ec53a8c1dbe2c49a4e3f64cb: Status 404 returned error can't find the container with id ea67f9d30541ec18b674180717367f12a638d6a1ec53a8c1dbe2c49a4e3f64cb Jul 16 12:17:00.001325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:00.001290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" event={"ID":"78449c1d-cfda-43fd-986f-240ef71706ec","Type":"ContainerStarted","Data":"ea67f9d30541ec18b674180717367f12a638d6a1ec53a8c1dbe2c49a4e3f64cb"} Jul 16 12:17:02.012116 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:02.012078 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" event={"ID":"78449c1d-cfda-43fd-986f-240ef71706ec","Type":"ContainerStarted","Data":"6e3db7e316ef36c7ace807b6eba2a5900d84becba424a4b4d325717ac44a4b4b"} Jul 16 12:17:02.012511 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:02.012169 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:02.032270 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:02.032222 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" podStartSLOduration=2.25488871 podStartE2EDuration="4.032207439s" totalImagesPullingTime="1.777318729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:02.030765003 +0000 UTC m=+619.724871741" watchObservedRunningTime="2026-07-16 12:17:02.032207439 +0000 UTC m=+619.726314157" Jul 16 12:17:03.853575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:03.853540 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g"] Jul 16 12:17:03.856065 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:03.856046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:03.858625 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:03.858604 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-76kjr\"" Jul 16 12:17:03.869563 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:03.869539 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g"] Jul 16 12:17:03.981218 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:03.981181 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5q2\" (UniqueName: \"kubernetes.io/projected/1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc-kube-api-access-tw5q2\") pod \"authorino-operator-75cc8d5cb7-9rs8g\" (UID: \"1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:04.082007 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:04.081966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5q2\" (UniqueName: \"kubernetes.io/projected/1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc-kube-api-access-tw5q2\") pod \"authorino-operator-75cc8d5cb7-9rs8g\" (UID: \"1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:04.096225 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:04.096188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5q2\" (UniqueName: \"kubernetes.io/projected/1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc-kube-api-access-tw5q2\") pod \"authorino-operator-75cc8d5cb7-9rs8g\" (UID: \"1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:04.165487 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:04.165451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:04.314817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:04.314788 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g"] Jul 16 12:17:04.317679 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:04.317651 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c81d8a4_14fb_474a_bd82_6d7fa5baf0bc.slice/crio-c6bd0d340ffd1924ac736924f934106d0003071e0238a931dea01d6abb4ef2c2 WatchSource:0}: Error finding container c6bd0d340ffd1924ac736924f934106d0003071e0238a931dea01d6abb4ef2c2: Status 404 returned error can't find the container with id c6bd0d340ffd1924ac736924f934106d0003071e0238a931dea01d6abb4ef2c2 Jul 16 12:17:05.025072 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:05.025036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" event={"ID":"1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc","Type":"ContainerStarted","Data":"c6bd0d340ffd1924ac736924f934106d0003071e0238a931dea01d6abb4ef2c2"} Jul 16 12:17:07.035077 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:07.035042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" event={"ID":"1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc","Type":"ContainerStarted","Data":"c83bd83d798f8087dafe11bfe716de4ae015b3c6911d3a3b30ba649e5f4925a0"} Jul 16 12:17:07.035370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:07.035157 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:07.055781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:07.055740 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" podStartSLOduration=1.416298486 podStartE2EDuration="4.05572466s" totalImagesPullingTime="2.639426174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:07.054848228 +0000 UTC m=+624.748954947" watchObservedRunningTime="2026-07-16 12:17:07.05572466 +0000 UTC m=+624.749831379" Jul 16 12:17:08.008809 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.008775 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2"] Jul 16 12:17:08.010889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.010873 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:08.013657 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:08.013630 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-operator-controller-env\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" type="*v1.ConfigMap" Jul 16 12:17:08.015211 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:08.015190 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-operator-controller-manager-864dd78c4c-j4pq2\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="9e850581-6bce-4f18-90e8-e1f2e4026069" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:08.015287 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:08.015208 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-operator-controller-manager-dockercfg-bq76r\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bq76r\"" type="*v1.Secret" Jul 16 12:17:08.046378 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.046353 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2"] Jul 16 12:17:08.119884 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.119858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6ft\" (UniqueName: \"kubernetes.io/projected/9e850581-6bce-4f18-90e8-e1f2e4026069-kube-api-access-kv6ft\") pod \"dns-operator-controller-manager-864dd78c4c-j4pq2\" (UID: \"9e850581-6bce-4f18-90e8-e1f2e4026069\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:08.220566 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.220539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6ft\" (UniqueName: \"kubernetes.io/projected/9e850581-6bce-4f18-90e8-e1f2e4026069-kube-api-access-kv6ft\") pod \"dns-operator-controller-manager-864dd78c4c-j4pq2\" (UID: \"9e850581-6bce-4f18-90e8-e1f2e4026069\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:08.238338 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:08.238310 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6ft\" (UniqueName: \"kubernetes.io/projected/9e850581-6bce-4f18-90e8-e1f2e4026069-kube-api-access-kv6ft\") pod \"dns-operator-controller-manager-864dd78c4c-j4pq2\" (UID: \"9e850581-6bce-4f18-90e8-e1f2e4026069\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:09.321028 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:09.320979 2583 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" secret="" err="failed to sync secret cache: timed out waiting for the condition" Jul 16 12:17:09.321411 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:09.321068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:09.343619 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:09.343593 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bq76r\"" Jul 16 12:17:09.445521 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:09.445492 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 12:17:09.449975 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:09.449944 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2"] Jul 16 12:17:09.452098 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:09.452073 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e850581_6bce_4f18_90e8_e1f2e4026069.slice/crio-11a2d5c6d123fc188ea9f7ae0b20d932fecef51027057e1f8c2b3817bc47b325 WatchSource:0}: Error finding container 11a2d5c6d123fc188ea9f7ae0b20d932fecef51027057e1f8c2b3817bc47b325: Status 404 returned error can't find the container with id 11a2d5c6d123fc188ea9f7ae0b20d932fecef51027057e1f8c2b3817bc47b325 Jul 16 12:17:10.049651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:10.049615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" event={"ID":"9e850581-6bce-4f18-90e8-e1f2e4026069","Type":"ContainerStarted","Data":"11a2d5c6d123fc188ea9f7ae0b20d932fecef51027057e1f8c2b3817bc47b325"} Jul 16 12:17:13.017625 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:13.017597 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:13.069101 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:13.069059 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" event={"ID":"9e850581-6bce-4f18-90e8-e1f2e4026069","Type":"ContainerStarted","Data":"14c2bef46e8648e6c962d0e1e42f2b04d3e44901c50d40b18fbd4914b065bde8"} Jul 16 12:17:13.069246 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:13.069206 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:13.126760 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:13.126715 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" podStartSLOduration=3.436061644 podStartE2EDuration="6.126701378s" totalImagesPullingTime="2.690639734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:13.125865098 +0000 UTC m=+630.819971820" watchObservedRunningTime="2026-07-16 12:17:13.126701378 +0000 UTC m=+630.820808099" Jul 16 12:17:18.041475 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:18.041446 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9rs8g" Jul 16 12:17:23.413909 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.413867 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk"] Jul 16 12:17:23.417340 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.417317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.420051 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.420027 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 12:17:23.421157 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.421140 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 12:17:23.421242 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.421199 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tf89p\"" Jul 16 12:17:23.425333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.425312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7db5f6-202b-4e57-92a3-35a1a6472546-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.425427 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.425363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7db5f6-202b-4e57-92a3-35a1a6472546-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.425488 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.425447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzkf\" (UniqueName: \"kubernetes.io/projected/0f7db5f6-202b-4e57-92a3-35a1a6472546-kube-api-access-5dzkf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.440144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.440124 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk"] Jul 16 12:17:23.526336 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.526302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzkf\" (UniqueName: \"kubernetes.io/projected/0f7db5f6-202b-4e57-92a3-35a1a6472546-kube-api-access-5dzkf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.526505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.526344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7db5f6-202b-4e57-92a3-35a1a6472546-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.526505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.526382 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7db5f6-202b-4e57-92a3-35a1a6472546-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.526954 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.526936 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7db5f6-202b-4e57-92a3-35a1a6472546-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.528913 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.528891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7db5f6-202b-4e57-92a3-35a1a6472546-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.547400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.547374 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzkf\" (UniqueName: \"kubernetes.io/projected/0f7db5f6-202b-4e57-92a3-35a1a6472546-kube-api-access-5dzkf\") pod \"kuadrant-console-plugin-66b5774985-nsqtk\" (UID: \"0f7db5f6-202b-4e57-92a3-35a1a6472546\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.726417 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.726319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" Jul 16 12:17:23.856165 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:23.856128 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk"] Jul 16 12:17:23.858936 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:23.858908 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f7db5f6_202b_4e57_92a3_35a1a6472546.slice/crio-b1e0ea7184f06102b8ffbb73b165417380b007aa99ed2bf1c910146e4d260ba6 WatchSource:0}: Error finding container b1e0ea7184f06102b8ffbb73b165417380b007aa99ed2bf1c910146e4d260ba6: Status 404 returned error can't find the container with id b1e0ea7184f06102b8ffbb73b165417380b007aa99ed2bf1c910146e4d260ba6 Jul 16 12:17:24.075531 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:24.075446 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-j4pq2" Jul 16 12:17:24.118917 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:24.118880 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" event={"ID":"0f7db5f6-202b-4e57-92a3-35a1a6472546","Type":"ContainerStarted","Data":"b1e0ea7184f06102b8ffbb73b165417380b007aa99ed2bf1c910146e4d260ba6"} Jul 16 12:17:33.067352 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.067285 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs"] Jul 16 12:17:33.073712 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.073683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.076606 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.076582 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w7v5k\"" Jul 16 12:17:33.096684 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.096648 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs"] Jul 16 12:17:33.096967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.096836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.097104 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.097024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhvc\" (UniqueName: \"kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.198405 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.198369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.198601 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.198419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhvc\" (UniqueName: \"kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.198827 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.198801 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.208769 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.208732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhvc\" (UniqueName: \"kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.390946 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.390907 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:33.575767 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.575735 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs"] Jul 16 12:17:33.577257 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:33.577226 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9c372e_2549_45ed_86ef_cd9fb72cfa14.slice/crio-a5eb19b0a10405e7e4a5317e8a6f31fffe602c54e0110321ea3dd2af0e733b32 WatchSource:0}: Error finding container a5eb19b0a10405e7e4a5317e8a6f31fffe602c54e0110321ea3dd2af0e733b32: Status 404 returned error can't find the container with id a5eb19b0a10405e7e4a5317e8a6f31fffe602c54e0110321ea3dd2af0e733b32 Jul 16 12:17:33.969838 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.969800 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs"] Jul 16 12:17:33.980463 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:33.980434 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs"] Jul 16 12:17:34.004535 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.004498 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:17:34.012344 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.012313 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.029475 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.029002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:17:34.031302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.031266 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt"] Jul 16 12:17:34.031588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.031534 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" containerName="manager" containerID="cri-o://6e3db7e316ef36c7ace807b6eba2a5900d84becba424a4b4d325717ac44a4b4b" gracePeriod=2 Jul 16 12:17:34.051843 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.051764 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt"] Jul 16 12:17:34.062363 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:34.062318 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:34.080656 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.080618 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc"] Jul 16 12:17:34.081164 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.081091 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" containerName="manager" Jul 16 12:17:34.081164 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.081110 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" containerName="manager" Jul 16 12:17:34.081294 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.081195 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" containerName="manager" Jul 16 12:17:34.084511 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.084469 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:34.088632 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:34.088587 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:34.104377 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.104326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2cwg\" (UniqueName: \"kubernetes.io/projected/bee8e9ae-f107-4d83-a256-62f367dd518a-kube-api-access-c2cwg\") pod \"limitador-operator-controller-manager-859d7b78bd-d9mhc\" (UID: \"bee8e9ae-f107-4d83-a256-62f367dd518a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:34.109473 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.108721 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc"] Jul 16 12:17:34.205380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.205340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jnwj\" (UniqueName: \"kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.205580 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.205405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cwg\" (UniqueName: \"kubernetes.io/projected/bee8e9ae-f107-4d83-a256-62f367dd518a-kube-api-access-c2cwg\") pod \"limitador-operator-controller-manager-859d7b78bd-d9mhc\" (UID: \"bee8e9ae-f107-4d83-a256-62f367dd518a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:34.205580 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.205455 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.223333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.223256 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cwg\" (UniqueName: \"kubernetes.io/projected/bee8e9ae-f107-4d83-a256-62f367dd518a-kube-api-access-c2cwg\") pod \"limitador-operator-controller-manager-859d7b78bd-d9mhc\" (UID: \"bee8e9ae-f107-4d83-a256-62f367dd518a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:34.306661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.306612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jnwj\" (UniqueName: \"kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.306844 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.306730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.307179 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.307150 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.337184 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.337149 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jnwj\" (UniqueName: \"kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-h96r5\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:34.414775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.414730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:34.628392 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:34.628355 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:48.246231 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.246074 2583 generic.go:356] "Generic (PLEG): container finished" podID="78449c1d-cfda-43fd-986f-240ef71706ec" containerID="6e3db7e316ef36c7ace807b6eba2a5900d84becba424a4b4d325717ac44a4b4b" exitCode=0 Jul 16 12:17:48.299385 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.299357 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:48.302763 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:48.302724 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:48.306408 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.306355 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:17:48.309523 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:48.309500 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bf69dc_81f0_46aa_91ef_cdce32ec6a88.slice/crio-e164e3f35affe97eb7583c686c9f10e54342c9f23185107d31943a694fff68b5 WatchSource:0}: Error finding container e164e3f35affe97eb7583c686c9f10e54342c9f23185107d31943a694fff68b5: Status 404 returned error can't find the container with id e164e3f35affe97eb7583c686c9f10e54342c9f23185107d31943a694fff68b5 Jul 16 12:17:48.328470 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.328439 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dhr4\" (UniqueName: \"kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4\") pod \"78449c1d-cfda-43fd-986f-240ef71706ec\" (UID: \"78449c1d-cfda-43fd-986f-240ef71706ec\") " Jul 16 12:17:48.330819 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.330787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4" (OuterVolumeSpecName: "kube-api-access-4dhr4") pod "78449c1d-cfda-43fd-986f-240ef71706ec" (UID: "78449c1d-cfda-43fd-986f-240ef71706ec"). InnerVolumeSpecName "kube-api-access-4dhr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:17:48.331403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.331381 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc"] Jul 16 12:17:48.334353 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:17:48.334331 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8e9ae_f107_4d83_a256_62f367dd518a.slice/crio-61f941c26531a73b8132c701c427ddfaa903cb7f2a3ccbef0984d5d3eb0115a6 WatchSource:0}: Error finding container 61f941c26531a73b8132c701c427ddfaa903cb7f2a3ccbef0984d5d3eb0115a6: Status 404 returned error can't find the container with id 61f941c26531a73b8132c701c427ddfaa903cb7f2a3ccbef0984d5d3eb0115a6 Jul 16 12:17:48.430260 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.430226 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhr4\" (UniqueName: \"kubernetes.io/projected/78449c1d-cfda-43fd-986f-240ef71706ec-kube-api-access-4dhr4\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:17:48.794002 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:48.793950 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" path="/var/lib/kubelet/pods/78449c1d-cfda-43fd-986f-240ef71706ec/volumes" Jul 16 12:17:49.254432 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.254387 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" event={"ID":"bee8e9ae-f107-4d83-a256-62f367dd518a","Type":"ContainerStarted","Data":"1925ef86d26f2be9ff4a6520533fd285ae0ed7eebe7e6ec07db7c1689050ecdb"} Jul 16 12:17:49.254898 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.254532 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:17:49.254898 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.254562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" event={"ID":"bee8e9ae-f107-4d83-a256-62f367dd518a","Type":"ContainerStarted","Data":"61f941c26531a73b8132c701c427ddfaa903cb7f2a3ccbef0984d5d3eb0115a6"} Jul 16 12:17:49.256164 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.256135 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" event={"ID":"17bf69dc-81f0-46aa-91ef-cdce32ec6a88","Type":"ContainerStarted","Data":"e164e3f35affe97eb7583c686c9f10e54342c9f23185107d31943a694fff68b5"} Jul 16 12:17:49.257692 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.257624 2583 scope.go:117] "RemoveContainer" containerID="6e3db7e316ef36c7ace807b6eba2a5900d84becba424a4b4d325717ac44a4b4b" Jul 16 12:17:49.257829 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.257779 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:49.261179 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.261128 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" event={"ID":"0f7db5f6-202b-4e57-92a3-35a1a6472546","Type":"ContainerStarted","Data":"53d49446ea6cce2b0de9d7612eadf355eefdfead5fb1dca2d5aedfeb21ad295f"} Jul 16 12:17:49.279923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.279885 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" podStartSLOduration=15.279870443 podStartE2EDuration="15.279870443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:49.27888942 +0000 UTC m=+666.972996174" watchObservedRunningTime="2026-07-16 12:17:49.279870443 +0000 UTC m=+666.973977165" Jul 16 12:17:49.281177 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:49.281132 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hwjkt\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="78449c1d-cfda-43fd-986f-240ef71706ec" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hwjkt" Jul 16 12:17:49.303400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:49.303362 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nsqtk" podStartSLOduration=1.923786285 podStartE2EDuration="26.303350161s" totalImagesPullingTime="24.379563876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:49.300870264 +0000 UTC m=+666.994976981" watchObservedRunningTime="2026-07-16 12:17:49.303350161 +0000 UTC m=+666.997456879" Jul 16 12:17:53.283067 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.283031 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" containerName="manager" containerID="cri-o://1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b" gracePeriod=2 Jul 16 12:17:53.284739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.284705 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" event={"ID":"17bf69dc-81f0-46aa-91ef-cdce32ec6a88","Type":"ContainerStarted","Data":"599588e6d3fdadc349bfe5102800c8d89d77401e1a0d39d1614c5ae6e6de6cea"} Jul 16 12:17:53.284868 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.284823 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:17:53.286378 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:53.286353 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:53.323438 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:53.323412 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:53.323683 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.323649 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" podStartSLOduration=16.345137839 podStartE2EDuration="20.323632422s" totalImagesPullingTime="3.978494583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:17:53.321328413 +0000 UTC m=+671.015435131" watchObservedRunningTime="2026-07-16 12:17:53.323632422 +0000 UTC m=+671.017739141" Jul 16 12:17:53.518532 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.518509 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:53.521302 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:53.521275 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:53.577793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.577734 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume\") pod \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " Jul 16 12:17:53.577979 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.577799 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nhvc\" (UniqueName: \"kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc\") pod \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\" (UID: \"6b9c372e-2549-45ed-86ef-cd9fb72cfa14\") " Jul 16 12:17:53.577979 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.577964 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b9c372e-2549-45ed-86ef-cd9fb72cfa14" (UID: "6b9c372e-2549-45ed-86ef-cd9fb72cfa14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:17:53.578096 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.578080 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-extensions-socket-volume\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:17:53.579637 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.579614 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc" (OuterVolumeSpecName: "kube-api-access-4nhvc") pod "6b9c372e-2549-45ed-86ef-cd9fb72cfa14" (UID: "6b9c372e-2549-45ed-86ef-cd9fb72cfa14"). InnerVolumeSpecName "kube-api-access-4nhvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:17:53.678676 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:53.678655 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhvc\" (UniqueName: \"kubernetes.io/projected/6b9c372e-2549-45ed-86ef-cd9fb72cfa14-kube-api-access-4nhvc\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:17:54.290768 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.290736 2583 generic.go:356] "Generic (PLEG): container finished" podID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" containerID="1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b" exitCode=2 Jul 16 12:17:54.291223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.290782 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:54.291223 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.290849 2583 scope.go:117] "RemoveContainer" containerID="1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b" Jul 16 12:17:54.293266 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:54.293230 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:54.300176 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.300155 2583 scope.go:117] "RemoveContainer" containerID="1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b" Jul 16 12:17:54.300448 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:54.300428 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b\": container with ID starting with 1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b not found: ID does not exist" containerID="1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b" Jul 16 12:17:54.300506 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.300455 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b"} err="failed to get container status \"1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b\": rpc error: code = NotFound desc = could not find container \"1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b\": container with ID starting with 1d8bce8c0a13f52a75a348d368241f5e7c78c44205c215aa5a81f98f4893590b not found: ID does not exist" Jul 16 12:17:54.301590 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:17:54.301565 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kp5bs\" is forbidden: User \"system:node:ip-10-0-139-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-22.ec2.internal' and this object" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kp5bs" Jul 16 12:17:54.792867 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:17:54.792840 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" path="/var/lib/kubelet/pods/6b9c372e-2549-45ed-86ef-cd9fb72cfa14/volumes" Jul 16 12:18:00.269518 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:00.269489 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d9mhc" Jul 16 12:18:04.293937 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:04.293904 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:18:17.221268 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.221234 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:18:17.221748 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.221423 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" podUID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" containerName="manager" containerID="cri-o://599588e6d3fdadc349bfe5102800c8d89d77401e1a0d39d1614c5ae6e6de6cea" gracePeriod=10 Jul 16 12:18:17.393618 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.393583 2583 generic.go:356] "Generic (PLEG): container finished" podID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" containerID="599588e6d3fdadc349bfe5102800c8d89d77401e1a0d39d1614c5ae6e6de6cea" exitCode=0 Jul 16 12:18:17.393762 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.393617 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" event={"ID":"17bf69dc-81f0-46aa-91ef-cdce32ec6a88","Type":"ContainerDied","Data":"599588e6d3fdadc349bfe5102800c8d89d77401e1a0d39d1614c5ae6e6de6cea"} Jul 16 12:18:17.468394 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.468372 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:18:17.552878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.552819 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume\") pod \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " Jul 16 12:18:17.552878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.552865 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jnwj\" (UniqueName: \"kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj\") pod \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\" (UID: \"17bf69dc-81f0-46aa-91ef-cdce32ec6a88\") " Jul 16 12:18:17.553270 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.553248 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17bf69dc-81f0-46aa-91ef-cdce32ec6a88" (UID: "17bf69dc-81f0-46aa-91ef-cdce32ec6a88"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:18:17.554839 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.554818 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj" (OuterVolumeSpecName: "kube-api-access-4jnwj") pod "17bf69dc-81f0-46aa-91ef-cdce32ec6a88" (UID: "17bf69dc-81f0-46aa-91ef-cdce32ec6a88"). InnerVolumeSpecName "kube-api-access-4jnwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:18:17.653458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.653431 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-extensions-socket-volume\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:18:17.653458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:17.653454 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jnwj\" (UniqueName: \"kubernetes.io/projected/17bf69dc-81f0-46aa-91ef-cdce32ec6a88-kube-api-access-4jnwj\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:18:18.404102 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.404072 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" Jul 16 12:18:18.404102 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.404080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5" event={"ID":"17bf69dc-81f0-46aa-91ef-cdce32ec6a88","Type":"ContainerDied","Data":"e164e3f35affe97eb7583c686c9f10e54342c9f23185107d31943a694fff68b5"} Jul 16 12:18:18.404670 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.404172 2583 scope.go:117] "RemoveContainer" containerID="599588e6d3fdadc349bfe5102800c8d89d77401e1a0d39d1614c5ae6e6de6cea" Jul 16 12:18:18.427946 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.427920 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:18:18.431050 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.431030 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-h96r5"] Jul 16 12:18:18.793608 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:18.793539 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" path="/var/lib/kubelet/pods/17bf69dc-81f0-46aa-91ef-cdce32ec6a88/volumes" Jul 16 12:18:21.032041 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032010 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5"] Jul 16 12:18:21.032386 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032329 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" containerName="manager" Jul 16 12:18:21.032386 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032342 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" containerName="manager" Jul 16 12:18:21.032386 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032368 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" containerName="manager" Jul 16 12:18:21.032386 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032374 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" containerName="manager" Jul 16 12:18:21.032523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032424 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17bf69dc-81f0-46aa-91ef-cdce32ec6a88" containerName="manager" Jul 16 12:18:21.032523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.032437 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b9c372e-2549-45ed-86ef-cd9fb72cfa14" containerName="manager" Jul 16 12:18:21.089826 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.089787 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5"] Jul 16 12:18:21.090012 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.089901 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.099213 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.099193 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2mlzg\"" Jul 16 12:18:21.176578 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176539 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176578 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72tj\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-kube-api-access-f72tj\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.176788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.176769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277677 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277677 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72tj\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-kube-api-access-f72tj\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.277907 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.277938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.278081 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278207 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.278118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.278235 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.278312 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.278605 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.278584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.280224 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.280203 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.280454 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.280419 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.291495 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.291444 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.291634 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.291613 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72tj\" (UniqueName: \"kubernetes.io/projected/dfbdeff0-9200-4c94-b0a2-845ecff1fa96-kube-api-access-f72tj\") pod \"maas-default-gateway-openshift-default-56654dcf7-xv9d5\" (UID: \"dfbdeff0-9200-4c94-b0a2-845ecff1fa96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.399269 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.399239 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:21.556723 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.556649 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5"] Jul 16 12:18:21.561646 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:18:21.561619 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbdeff0_9200_4c94_b0a2_845ecff1fa96.slice/crio-373ce5557ed32ffb1d72390619ef0d170b4fd601fc853e683d33b01f7ac3510d WatchSource:0}: Error finding container 373ce5557ed32ffb1d72390619ef0d170b4fd601fc853e683d33b01f7ac3510d: Status 404 returned error can't find the container with id 373ce5557ed32ffb1d72390619ef0d170b4fd601fc853e683d33b01f7ac3510d Jul 16 12:18:21.563679 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.563646 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:18:21.563773 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.563707 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:18:21.563773 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:21.563734 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:18:22.424533 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:22.424499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" event={"ID":"dfbdeff0-9200-4c94-b0a2-845ecff1fa96","Type":"ContainerStarted","Data":"8f39a71543c0a521c62deba7a40f998166cd5fcf2ca6de6d56f3ab8b18ea6dee"} Jul 16 12:18:22.424897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:22.424574 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" event={"ID":"dfbdeff0-9200-4c94-b0a2-845ecff1fa96","Type":"ContainerStarted","Data":"373ce5557ed32ffb1d72390619ef0d170b4fd601fc853e683d33b01f7ac3510d"} Jul 16 12:18:22.468057 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:22.468017 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" podStartSLOduration=2.467982219 podStartE2EDuration="2.467982219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:22.467851107 +0000 UTC m=+700.161957825" watchObservedRunningTime="2026-07-16 12:18:22.467982219 +0000 UTC m=+700.162088938" Jul 16 12:18:23.400296 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.400261 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:23.404958 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.404931 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:23.428680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.428658 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:23.429300 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.429284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:18:23.816000 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.815911 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:23.819462 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.819444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:23.822054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.822040 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 12:18:23.833789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.833763 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:23.898770 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.898741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:23.898915 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.898803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbwg\" (UniqueName: \"kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:23.910716 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.910691 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:23.999741 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.999706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:23.999879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:23.999773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbwg\" (UniqueName: \"kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:24.000411 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:24.000392 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:24.012463 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:24.012437 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbwg\" (UniqueName: \"kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg\") pod \"limitador-limitador-64c94fd64f-bh5kf\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:24.129717 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:24.129695 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:24.263036 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:24.262925 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:24.265714 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:18:24.265683 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcff3c19_7f92_43a4_84ff_4b3734413165.slice/crio-51e0e9da7ef6f4499270965e0faf14037288cf61d2076516565973e748af2cbf WatchSource:0}: Error finding container 51e0e9da7ef6f4499270965e0faf14037288cf61d2076516565973e748af2cbf: Status 404 returned error can't find the container with id 51e0e9da7ef6f4499270965e0faf14037288cf61d2076516565973e748af2cbf Jul 16 12:18:24.433552 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:24.433473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" event={"ID":"dcff3c19-7f92-43a4-84ff-4b3734413165","Type":"ContainerStarted","Data":"51e0e9da7ef6f4499270965e0faf14037288cf61d2076516565973e748af2cbf"} Jul 16 12:18:28.462372 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:28.462335 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" event={"ID":"dcff3c19-7f92-43a4-84ff-4b3734413165","Type":"ContainerStarted","Data":"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7"} Jul 16 12:18:28.462820 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:28.462463 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:28.495665 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:28.495624 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" podStartSLOduration=2.166070005 podStartE2EDuration="5.495610909s" totalImagesPullingTime="3.329540904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:28.495090444 +0000 UTC m=+706.189197197" watchObservedRunningTime="2026-07-16 12:18:28.495610909 +0000 UTC m=+706.189717627" Jul 16 12:18:39.467459 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:39.467382 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:39.563149 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:39.563111 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:39.563331 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:39.563288 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" podUID="dcff3c19-7f92-43a4-84ff-4b3734413165" containerName="limitador" containerID="cri-o://850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7" gracePeriod=30 Jul 16 12:18:40.106330 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.106308 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:40.225350 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.225270 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmbwg\" (UniqueName: \"kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg\") pod \"dcff3c19-7f92-43a4-84ff-4b3734413165\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " Jul 16 12:18:40.225506 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.225358 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file\") pod \"dcff3c19-7f92-43a4-84ff-4b3734413165\" (UID: \"dcff3c19-7f92-43a4-84ff-4b3734413165\") " Jul 16 12:18:40.225655 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.225632 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file" (OuterVolumeSpecName: "config-file") pod "dcff3c19-7f92-43a4-84ff-4b3734413165" (UID: "dcff3c19-7f92-43a4-84ff-4b3734413165"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:18:40.227274 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.227254 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg" (OuterVolumeSpecName: "kube-api-access-kmbwg") pod "dcff3c19-7f92-43a4-84ff-4b3734413165" (UID: "dcff3c19-7f92-43a4-84ff-4b3734413165"). InnerVolumeSpecName "kube-api-access-kmbwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:18:40.326438 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.326403 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbwg\" (UniqueName: \"kubernetes.io/projected/dcff3c19-7f92-43a4-84ff-4b3734413165-kube-api-access-kmbwg\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:18:40.326438 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.326430 2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dcff3c19-7f92-43a4-84ff-4b3734413165-config-file\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:18:40.515057 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.514955 2583 generic.go:356] "Generic (PLEG): container finished" podID="dcff3c19-7f92-43a4-84ff-4b3734413165" containerID="850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7" exitCode=0 Jul 16 12:18:40.515057 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.515012 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" Jul 16 12:18:40.515057 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.515037 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" event={"ID":"dcff3c19-7f92-43a4-84ff-4b3734413165","Type":"ContainerDied","Data":"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7"} Jul 16 12:18:40.515521 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.515106 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bh5kf" event={"ID":"dcff3c19-7f92-43a4-84ff-4b3734413165","Type":"ContainerDied","Data":"51e0e9da7ef6f4499270965e0faf14037288cf61d2076516565973e748af2cbf"} Jul 16 12:18:40.515521 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.515137 2583 scope.go:117] "RemoveContainer" containerID="850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7" Jul 16 12:18:40.523700 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.523684 2583 scope.go:117] "RemoveContainer" containerID="850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7" Jul 16 12:18:40.523941 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:18:40.523924 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7\": container with ID starting with 850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7 not found: ID does not exist" containerID="850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7" Jul 16 12:18:40.523983 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.523948 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7"} err="failed to get container status \"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7\": rpc error: code = NotFound desc = could not find container \"850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7\": container with ID starting with 850ad7d990adefc1d6d10dfb48816f7d6b38f6e30a35c642f73746748de9c2b7 not found: ID does not exist" Jul 16 12:18:40.538589 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.538568 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:40.545059 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.545040 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bh5kf"] Jul 16 12:18:40.793373 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:40.793299 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcff3c19-7f92-43a4-84ff-4b3734413165" path="/var/lib/kubelet/pods/dcff3c19-7f92-43a4-84ff-4b3734413165/volumes" Jul 16 12:18:46.033360 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.033326 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99tv7"] Jul 16 12:18:46.033747 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.033670 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcff3c19-7f92-43a4-84ff-4b3734413165" containerName="limitador" Jul 16 12:18:46.033747 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.033681 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcff3c19-7f92-43a4-84ff-4b3734413165" containerName="limitador" Jul 16 12:18:46.033747 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.033731 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcff3c19-7f92-43a4-84ff-4b3734413165" containerName="limitador" Jul 16 12:18:46.036976 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.036954 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.051881 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.051854 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j5nq4\"" Jul 16 12:18:46.054723 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.054705 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 12:18:46.060356 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.060335 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 12:18:46.072416 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.072383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx9k\" (UniqueName: \"kubernetes.io/projected/ec31fbdb-59c7-4502-a187-7a3976e75ed7-kube-api-access-xvx9k\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.072583 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.072481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec31fbdb-59c7-4502-a187-7a3976e75ed7-data\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.079641 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.079621 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 16 12:18:46.160085 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.160051 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99tv7"] Jul 16 12:18:46.173646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.173617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec31fbdb-59c7-4502-a187-7a3976e75ed7-data\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.173800 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.173675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx9k\" (UniqueName: \"kubernetes.io/projected/ec31fbdb-59c7-4502-a187-7a3976e75ed7-kube-api-access-xvx9k\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.174093 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.174075 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec31fbdb-59c7-4502-a187-7a3976e75ed7-data\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.187712 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.187684 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx9k\" (UniqueName: \"kubernetes.io/projected/ec31fbdb-59c7-4502-a187-7a3976e75ed7-kube-api-access-xvx9k\") pod \"postgres-69fb4594cf-99tv7\" (UID: \"ec31fbdb-59c7-4502-a187-7a3976e75ed7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.346980 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.346885 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:46.501485 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.501454 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99tv7"] Jul 16 12:18:46.503870 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:18:46.503839 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec31fbdb_59c7_4502_a187_7a3976e75ed7.slice/crio-5f26e96c38d6d39da608b4e7669f3c11db676e799993e56e7ffc2c7450bc39e1 WatchSource:0}: Error finding container 5f26e96c38d6d39da608b4e7669f3c11db676e799993e56e7ffc2c7450bc39e1: Status 404 returned error can't find the container with id 5f26e96c38d6d39da608b4e7669f3c11db676e799993e56e7ffc2c7450bc39e1 Jul 16 12:18:46.542647 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:46.542612 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" event={"ID":"ec31fbdb-59c7-4502-a187-7a3976e75ed7","Type":"ContainerStarted","Data":"5f26e96c38d6d39da608b4e7669f3c11db676e799993e56e7ffc2c7450bc39e1"} Jul 16 12:18:51.570104 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:51.570053 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" event={"ID":"ec31fbdb-59c7-4502-a187-7a3976e75ed7","Type":"ContainerStarted","Data":"579798c43b525fac4983871648f526a726149b801a9e41f6d2569af02131a014"} Jul 16 12:18:51.570603 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:51.570122 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:18:51.593951 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:51.593892 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" podStartSLOduration=1.984678344 podStartE2EDuration="6.593875994s" totalImagesPullingTime="4.60919765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:18:51.593064882 +0000 UTC m=+729.287171598" watchObservedRunningTime="2026-07-16 12:18:51.593875994 +0000 UTC m=+729.287982712" Jul 16 12:18:57.605701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:18:57.605670 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99tv7" Jul 16 12:19:06.555891 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.555858 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:06.578268 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.578235 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:06.578447 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.578329 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.581271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.581249 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kkp9\"" Jul 16 12:19:06.581271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.581266 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 12:19:06.581455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.581287 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 12:19:06.643456 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.643428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kvv\" (UniqueName: \"kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.643618 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.643463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.744633 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.744593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kvv\" (UniqueName: \"kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.744811 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.744656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.744811 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:06.744785 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 16 12:19:06.744900 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:06.744858 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert podName:53f46198-d17c-4523-877c-1b666fe16236 nodeName:}" failed. No retries permitted until 2026-07-16 12:19:07.244843276 +0000 UTC m=+744.938949973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert") pod "maas-controller-564ccc6588-mtbdx" (UID: "53f46198-d17c-4523-877c-1b666fe16236") : secret "maas-controller-webhook-cert" not found Jul 16 12:19:06.747969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.747944 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:19:06.750328 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.750306 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:06.756103 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.756074 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kvv\" (UniqueName: \"kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:06.762724 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.762698 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:19:06.845329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.845227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:06.845329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.845275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cjf\" (UniqueName: \"kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:06.946191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.946148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:06.946396 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.946204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cjf\" (UniqueName: \"kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:06.946396 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:06.946301 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 16 12:19:06.946396 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:06.946371 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert podName:90623247-c507-49e8-b5ab-522bbdf6408b nodeName:}" failed. No retries permitted until 2026-07-16 12:19:07.446353768 +0000 UTC m=+745.140460469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert") pod "maas-controller-6dcdc7c8b8-lstfz" (UID: "90623247-c507-49e8-b5ab-522bbdf6408b") : secret "maas-controller-webhook-cert" not found Jul 16 12:19:06.955863 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:06.955831 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cjf\" (UniqueName: \"kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:07.247976 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.247935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:07.250318 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.250298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") pod \"maas-controller-564ccc6588-mtbdx\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:07.449944 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.449907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:07.452497 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.452464 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lstfz\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:07.489315 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.489276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:07.623088 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.623012 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:07.625291 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:19:07.625264 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f46198_d17c_4523_877c_1b666fe16236.slice/crio-1ccb666cbe62135ead45fcab8ca9371ad48be26c4a78a778529210ae2f95745b WatchSource:0}: Error finding container 1ccb666cbe62135ead45fcab8ca9371ad48be26c4a78a778529210ae2f95745b: Status 404 returned error can't find the container with id 1ccb666cbe62135ead45fcab8ca9371ad48be26c4a78a778529210ae2f95745b Jul 16 12:19:07.645054 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.645008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-564ccc6588-mtbdx" event={"ID":"53f46198-d17c-4523-877c-1b666fe16236","Type":"ContainerStarted","Data":"1ccb666cbe62135ead45fcab8ca9371ad48be26c4a78a778529210ae2f95745b"} Jul 16 12:19:07.670229 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.670196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:07.814718 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:07.814635 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:19:07.817478 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:19:07.817449 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90623247_c507_49e8_b5ab_522bbdf6408b.slice/crio-9b813b17a98df94a4d9152514c209012c4b948a5af69e6f448df7818f5c5ff0e WatchSource:0}: Error finding container 9b813b17a98df94a4d9152514c209012c4b948a5af69e6f448df7818f5c5ff0e: Status 404 returned error can't find the container with id 9b813b17a98df94a4d9152514c209012c4b948a5af69e6f448df7818f5c5ff0e Jul 16 12:19:08.653655 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:08.653616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" event={"ID":"90623247-c507-49e8-b5ab-522bbdf6408b","Type":"ContainerStarted","Data":"9b813b17a98df94a4d9152514c209012c4b948a5af69e6f448df7818f5c5ff0e"} Jul 16 12:19:10.666819 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.666775 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" event={"ID":"90623247-c507-49e8-b5ab-522bbdf6408b","Type":"ContainerStarted","Data":"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924"} Jul 16 12:19:10.667251 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.666843 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:10.668400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.668368 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-564ccc6588-mtbdx" event={"ID":"53f46198-d17c-4523-877c-1b666fe16236","Type":"ContainerStarted","Data":"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4"} Jul 16 12:19:10.668552 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.668431 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:10.698271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.698206 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" podStartSLOduration=2.012919893 podStartE2EDuration="4.698186838s" totalImagesPullingTime="2.685266945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:10.696617344 +0000 UTC m=+748.390724064" watchObservedRunningTime="2026-07-16 12:19:10.698186838 +0000 UTC m=+748.392293556" Jul 16 12:19:10.722012 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:10.721943 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-564ccc6588-mtbdx" podStartSLOduration=1.854240297 podStartE2EDuration="4.721928076s" totalImagesPullingTime="2.867687779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:10.719216547 +0000 UTC m=+748.413323293" watchObservedRunningTime="2026-07-16 12:19:10.721928076 +0000 UTC m=+748.416034796" Jul 16 12:19:21.675790 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:21.675760 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:19:21.676190 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:21.675967 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:21.748590 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:21.748552 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:21.748799 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:21.748770 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-564ccc6588-mtbdx" podUID="53f46198-d17c-4523-877c-1b666fe16236" containerName="manager" containerID="cri-o://52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4" gracePeriod=30 Jul 16 12:19:22.002808 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.002777 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:22.077977 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.077932 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54kvv\" (UniqueName: \"kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv\") pod \"53f46198-d17c-4523-877c-1b666fe16236\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " Jul 16 12:19:22.078170 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.078024 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") pod \"53f46198-d17c-4523-877c-1b666fe16236\" (UID: \"53f46198-d17c-4523-877c-1b666fe16236\") " Jul 16 12:19:22.080244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.080212 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "53f46198-d17c-4523-877c-1b666fe16236" (UID: "53f46198-d17c-4523-877c-1b666fe16236"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:19:22.080361 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.080315 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv" (OuterVolumeSpecName: "kube-api-access-54kvv") pod "53f46198-d17c-4523-877c-1b666fe16236" (UID: "53f46198-d17c-4523-877c-1b666fe16236"). InnerVolumeSpecName "kube-api-access-54kvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:19:22.178687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.178653 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f46198-d17c-4523-877c-1b666fe16236-webhook-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:19:22.178687 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.178682 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54kvv\" (UniqueName: \"kubernetes.io/projected/53f46198-d17c-4523-877c-1b666fe16236-kube-api-access-54kvv\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:19:22.726192 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.726148 2583 generic.go:356] "Generic (PLEG): container finished" podID="53f46198-d17c-4523-877c-1b666fe16236" containerID="52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4" exitCode=0 Jul 16 12:19:22.726585 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.726219 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-564ccc6588-mtbdx" Jul 16 12:19:22.726585 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.726226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-564ccc6588-mtbdx" event={"ID":"53f46198-d17c-4523-877c-1b666fe16236","Type":"ContainerDied","Data":"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4"} Jul 16 12:19:22.726585 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.726315 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-564ccc6588-mtbdx" event={"ID":"53f46198-d17c-4523-877c-1b666fe16236","Type":"ContainerDied","Data":"1ccb666cbe62135ead45fcab8ca9371ad48be26c4a78a778529210ae2f95745b"} Jul 16 12:19:22.726585 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.726344 2583 scope.go:117] "RemoveContainer" containerID="52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4" Jul 16 12:19:22.740515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.740493 2583 scope.go:117] "RemoveContainer" containerID="52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4" Jul 16 12:19:22.740791 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:22.740771 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4\": container with ID starting with 52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4 not found: ID does not exist" containerID="52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4" Jul 16 12:19:22.740864 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.740801 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4"} err="failed to get container status \"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4\": rpc error: code = NotFound desc = could not find container \"52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4\": container with ID starting with 52e5e955d5751cdb0602d9d25880796cc653ce87dd30f5e94b6799ca832fa3b4 not found: ID does not exist" Jul 16 12:19:22.750969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.750945 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:22.754814 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.754792 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-564ccc6588-mtbdx"] Jul 16 12:19:22.794285 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:22.794260 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f46198-d17c-4523-877c-1b666fe16236" path="/var/lib/kubelet/pods/53f46198-d17c-4523-877c-1b666fe16236/volumes" Jul 16 12:19:40.018086 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.018050 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d96948d97-pbz79"] Jul 16 12:19:40.018484 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.018461 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f46198-d17c-4523-877c-1b666fe16236" containerName="manager" Jul 16 12:19:40.018484 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.018474 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f46198-d17c-4523-877c-1b666fe16236" containerName="manager" Jul 16 12:19:40.018566 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.018555 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f46198-d17c-4523-877c-1b666fe16236" containerName="manager" Jul 16 12:19:40.020588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.020564 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.023388 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.023359 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x8mkx\"" Jul 16 12:19:40.023793 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.023774 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 16 12:19:40.037333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.037306 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d96948d97-pbz79"] Jul 16 12:19:40.137880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.137843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.137880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.137882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndbw\" (UniqueName: \"kubernetes.io/projected/a863f304-5004-47fd-8ac7-7ca10f4221b7-kube-api-access-rndbw\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.138106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.137905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a863f304-5004-47fd-8ac7-7ca10f4221b7-openshift-service-ca\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.239210 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.239176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.239400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.239224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndbw\" (UniqueName: \"kubernetes.io/projected/a863f304-5004-47fd-8ac7-7ca10f4221b7-kube-api-access-rndbw\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.239400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.239247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a863f304-5004-47fd-8ac7-7ca10f4221b7-openshift-service-ca\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.239400 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:40.239343 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 16 12:19:40.239564 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:19:40.239433 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls podName:a863f304-5004-47fd-8ac7-7ca10f4221b7 nodeName:}" failed. No retries permitted until 2026-07-16 12:19:40.739411422 +0000 UTC m=+778.433518137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls") pod "maas-api-7d96948d97-pbz79" (UID: "a863f304-5004-47fd-8ac7-7ca10f4221b7") : secret "maas-api-serving-cert" not found Jul 16 12:19:40.239852 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.239834 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a863f304-5004-47fd-8ac7-7ca10f4221b7-openshift-service-ca\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.248378 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.248353 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndbw\" (UniqueName: \"kubernetes.io/projected/a863f304-5004-47fd-8ac7-7ca10f4221b7-kube-api-access-rndbw\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.639790 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.639753 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hcw56"] Jul 16 12:19:40.642213 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.642195 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.645454 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.645431 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dc2jd\"" Jul 16 12:19:40.646567 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.646544 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 12:19:40.656094 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.656070 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hcw56"] Jul 16 12:19:40.743434 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.743395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsntx\" (UniqueName: \"kubernetes.io/projected/525e4580-d118-4b19-9b09-9417fc42c178-kube-api-access-rsntx\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.743627 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.743469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.743627 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.743542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525e4580-d118-4b19-9b09-9417fc42c178-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.745853 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.745826 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a863f304-5004-47fd-8ac7-7ca10f4221b7-maas-api-tls\") pod \"maas-api-7d96948d97-pbz79\" (UID: \"a863f304-5004-47fd-8ac7-7ca10f4221b7\") " pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.844759 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.844720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsntx\" (UniqueName: \"kubernetes.io/projected/525e4580-d118-4b19-9b09-9417fc42c178-kube-api-access-rsntx\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.844922 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.844815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525e4580-d118-4b19-9b09-9417fc42c178-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.845571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.845550 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525e4580-d118-4b19-9b09-9417fc42c178-plugins-config-volume\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.854322 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.854299 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsntx\" (UniqueName: \"kubernetes.io/projected/525e4580-d118-4b19-9b09-9417fc42c178-kube-api-access-rsntx\") pod \"payload-processing-5bf47cd988-hcw56\" (UID: \"525e4580-d118-4b19-9b09-9417fc42c178\") " pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:40.931397 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.931298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:40.953637 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:40.953608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:41.104147 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:41.104112 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d96948d97-pbz79"] Jul 16 12:19:41.108310 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:19:41.108275 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda863f304_5004_47fd_8ac7_7ca10f4221b7.slice/crio-e14aad2cec0ce04bd3ece3317b4e7b4c5b6e35b417ed02d9b51acf93a39a4bd9 WatchSource:0}: Error finding container e14aad2cec0ce04bd3ece3317b4e7b4c5b6e35b417ed02d9b51acf93a39a4bd9: Status 404 returned error can't find the container with id e14aad2cec0ce04bd3ece3317b4e7b4c5b6e35b417ed02d9b51acf93a39a4bd9 Jul 16 12:19:41.109865 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:41.109842 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:19:41.134396 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:41.134367 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-hcw56"] Jul 16 12:19:41.136536 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:19:41.136503 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525e4580_d118_4b19_9b09_9417fc42c178.slice/crio-612368be8ab549ab255fcd10949566c7cd29d898d923c790064b02ffa2c6d08f WatchSource:0}: Error finding container 612368be8ab549ab255fcd10949566c7cd29d898d923c790064b02ffa2c6d08f: Status 404 returned error can't find the container with id 612368be8ab549ab255fcd10949566c7cd29d898d923c790064b02ffa2c6d08f Jul 16 12:19:41.820488 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:41.820437 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" event={"ID":"525e4580-d118-4b19-9b09-9417fc42c178","Type":"ContainerStarted","Data":"612368be8ab549ab255fcd10949566c7cd29d898d923c790064b02ffa2c6d08f"} Jul 16 12:19:41.821641 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:41.821615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" event={"ID":"a863f304-5004-47fd-8ac7-7ca10f4221b7","Type":"ContainerStarted","Data":"e14aad2cec0ce04bd3ece3317b4e7b4c5b6e35b417ed02d9b51acf93a39a4bd9"} Jul 16 12:19:43.835973 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.835923 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" event={"ID":"525e4580-d118-4b19-9b09-9417fc42c178","Type":"ContainerStarted","Data":"65093189edf80ebec5c340837be01e871f90d3669d95d12c2e76f6858496859a"} Jul 16 12:19:43.836845 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.836818 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:43.838535 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.838500 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" event={"ID":"a863f304-5004-47fd-8ac7-7ca10f4221b7","Type":"ContainerStarted","Data":"3a0b2b00206647e2ef4cd47e740f3cc6f21b376c88fdcd630a63af36e63666f8"} Jul 16 12:19:43.839077 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.839052 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:43.855348 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.855288 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" podStartSLOduration=1.738415178 podStartE2EDuration="3.855270484s" totalImagesPullingTime="2.116855306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:43.854819152 +0000 UTC m=+781.548925870" watchObservedRunningTime="2026-07-16 12:19:43.855270484 +0000 UTC m=+781.549377203" Jul 16 12:19:43.876463 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:43.876409 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" podStartSLOduration=2.733465268 podStartE2EDuration="4.876394344s" totalImagesPullingTime="2.142929076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:19:43.875533289 +0000 UTC m=+781.569640040" watchObservedRunningTime="2026-07-16 12:19:43.876394344 +0000 UTC m=+781.570501063" Jul 16 12:19:50.862255 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.862225 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d96948d97-pbz79" Jul 16 12:19:50.879692 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.879664 2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbdeff0-9200-4c94-b0a2-845ecff1fa96" containerID="8f39a71543c0a521c62deba7a40f998166cd5fcf2ca6de6d56f3ab8b18ea6dee" exitCode=0 Jul 16 12:19:50.879832 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.879716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" event={"ID":"dfbdeff0-9200-4c94-b0a2-845ecff1fa96","Type":"ContainerDied","Data":"8f39a71543c0a521c62deba7a40f998166cd5fcf2ca6de6d56f3ab8b18ea6dee"} Jul 16 12:19:50.880089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.880074 2583 scope.go:117] "RemoveContainer" containerID="8f39a71543c0a521c62deba7a40f998166cd5fcf2ca6de6d56f3ab8b18ea6dee" Jul 16 12:19:50.881036 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.881011 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:19:50.881092 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.881077 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:19:50.881137 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:50.881125 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:19:51.886387 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:51.886345 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" event={"ID":"dfbdeff0-9200-4c94-b0a2-845ecff1fa96","Type":"ContainerStarted","Data":"140c9e062784cb7cc34416214c9730f23a2a46b1c9d4d6a040a34eac34577a0a"} Jul 16 12:19:52.400434 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:52.400398 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:19:52.401661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:52.401626 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xv9d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 16 12:19:52.401806 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:52.401690 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" podUID="dfbdeff0-9200-4c94-b0a2-845ecff1fa96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 16 12:19:53.399821 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:53.399786 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xv9d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 16 12:19:53.400329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:53.399842 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" podUID="dfbdeff0-9200-4c94-b0a2-845ecff1fa96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 16 12:19:53.429154 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:53.429113 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:19:54.399627 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:54.399586 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xv9d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 16 12:19:54.399809 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:54.399645 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" podUID="dfbdeff0-9200-4c94-b0a2-845ecff1fa96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 16 12:19:55.400280 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:55.400246 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xv9d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 16 12:19:55.400692 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:55.400302 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" podUID="dfbdeff0-9200-4c94-b0a2-845ecff1fa96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 16 12:19:55.858133 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:55.858033 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-hcw56" Jul 16 12:19:56.404437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:56.404403 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:19:56.405467 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:19:56.405443 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xv9d5" Jul 16 12:20:23.615939 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.615903 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj"] Jul 16 12:20:23.620552 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.620526 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.623032 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.623008 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 12:20:23.624330 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.624294 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 12:20:23.624330 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.624303 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 12:20:23.624497 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.624351 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hgfq8\"" Jul 16 12:20:23.632775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.632741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj"] Jul 16 12:20:23.717946 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.717914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.717946 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.717948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.718182 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.717966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.718182 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.718112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.718248 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.718175 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.718248 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.718213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgsl\" (UniqueName: \"kubernetes.io/projected/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-kube-api-access-2xgsl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819069 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819227 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819101 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819227 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819227 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgsl\" (UniqueName: \"kubernetes.io/projected/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-kube-api-access-2xgsl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819345 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819402 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819518 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819488 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819589 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819531 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.819711 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.819688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.821466 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.821443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.821846 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.821825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.830789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.830765 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgsl\" (UniqueName: \"kubernetes.io/projected/47b002ee-14bd-4ec5-919f-0b8c3e8fa677-kube-api-access-2xgsl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7z4sj\" (UID: \"47b002ee-14bd-4ec5-919f-0b8c3e8fa677\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:23.933170 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:23.933134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:24.091873 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:24.091847 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj"] Jul 16 12:20:24.093405 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:20:24.093373 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b002ee_14bd_4ec5_919f_0b8c3e8fa677.slice/crio-ccb53426e9a5a1bf80353a7301012e1cfd6bd06a077f2887995262eeaa445728 WatchSource:0}: Error finding container ccb53426e9a5a1bf80353a7301012e1cfd6bd06a077f2887995262eeaa445728: Status 404 returned error can't find the container with id ccb53426e9a5a1bf80353a7301012e1cfd6bd06a077f2887995262eeaa445728 Jul 16 12:20:25.035645 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:25.035609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" event={"ID":"47b002ee-14bd-4ec5-919f-0b8c3e8fa677","Type":"ContainerStarted","Data":"ccb53426e9a5a1bf80353a7301012e1cfd6bd06a077f2887995262eeaa445728"} Jul 16 12:20:30.501509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.501455 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc"] Jul 16 12:20:30.505735 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.505707 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.509264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.509236 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 12:20:30.519044 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.519015 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc"] Jul 16 12:20:30.581156 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.581156 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.581401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754ad6ca-f51a-4603-aae0-56f243b24ade-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.581401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.581401 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf65b\" (UniqueName: \"kubernetes.io/projected/754ad6ca-f51a-4603-aae0-56f243b24ade-kube-api-access-wf65b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.581549 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.581434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682195 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf65b\" (UniqueName: \"kubernetes.io/projected/754ad6ca-f51a-4603-aae0-56f243b24ade-kube-api-access-wf65b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682341 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754ad6ca-f51a-4603-aae0-56f243b24ade-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682589 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682718 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682693 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.682781 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.682753 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.684447 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.684418 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754ad6ca-f51a-4603-aae0-56f243b24ade-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.684817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.684798 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754ad6ca-f51a-4603-aae0-56f243b24ade-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.691371 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.691202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf65b\" (UniqueName: \"kubernetes.io/projected/754ad6ca-f51a-4603-aae0-56f243b24ade-kube-api-access-wf65b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qnxlc\" (UID: \"754ad6ca-f51a-4603-aae0-56f243b24ade\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.819325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.819222 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:30.958432 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:30.958372 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc"] Jul 16 12:20:30.961153 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:20:30.961127 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754ad6ca_f51a_4603_aae0_56f243b24ade.slice/crio-21b3849952e4c91a3ba4f502858e71c53f34ffc587cdf36e801ccbb12e550860 WatchSource:0}: Error finding container 21b3849952e4c91a3ba4f502858e71c53f34ffc587cdf36e801ccbb12e550860: Status 404 returned error can't find the container with id 21b3849952e4c91a3ba4f502858e71c53f34ffc587cdf36e801ccbb12e550860 Jul 16 12:20:31.063333 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:31.063295 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" event={"ID":"47b002ee-14bd-4ec5-919f-0b8c3e8fa677","Type":"ContainerStarted","Data":"dc3d672c06b050006edf0948a820ffe9842739eceea8ae3cb3bdf2c8d8fb1c62"} Jul 16 12:20:31.064495 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:31.064456 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" event={"ID":"754ad6ca-f51a-4603-aae0-56f243b24ade","Type":"ContainerStarted","Data":"21b3849952e4c91a3ba4f502858e71c53f34ffc587cdf36e801ccbb12e550860"} Jul 16 12:20:31.085972 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:31.085874 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" podStartSLOduration=1.548498393 podStartE2EDuration="8.085856995s" totalImagesPullingTime="6.537358602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:31.084579922 +0000 UTC m=+828.778686665" watchObservedRunningTime="2026-07-16 12:20:31.085856995 +0000 UTC m=+828.779963713" Jul 16 12:20:32.075163 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:32.075107 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" event={"ID":"754ad6ca-f51a-4603-aae0-56f243b24ade","Type":"ContainerStarted","Data":"5b1c40f24c8516f0f85ed537939fac256878cb9e90396ef810400ad58147d53a"} Jul 16 12:20:32.093886 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:32.093827 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" podStartSLOduration=1.891018808 podStartE2EDuration="2.093812273s" totalImagesPullingTime="202.793465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:32.093295541 +0000 UTC m=+829.787402283" watchObservedRunningTime="2026-07-16 12:20:32.093812273 +0000 UTC m=+829.787918991" Jul 16 12:20:33.933561 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:33.933526 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:33.934074 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:33.933612 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:33.945944 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:33.945919 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:34.095040 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:34.095006 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7z4sj" Jul 16 12:20:40.819712 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:40.819677 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:40.820231 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:40.819747 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:40.832508 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:40.832480 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:41.126923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:41.126896 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qnxlc" Jul 16 12:20:53.080204 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.080156 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b"] Jul 16 12:20:53.083929 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.083906 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.086575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.086555 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 12:20:53.093457 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.093436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b"] Jul 16 12:20:53.175369 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv46t\" (UniqueName: \"kubernetes.io/projected/e45db0f5-c580-4237-b82d-4670b51665f9-kube-api-access-nv46t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.175574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.175574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.175574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.175574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45db0f5-c580-4237-b82d-4670b51665f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.175574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.175555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.276622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.276622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv46t\" (UniqueName: \"kubernetes.io/projected/e45db0f5-c580-4237-b82d-4670b51665f9-kube-api-access-nv46t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.276861 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276682 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.276861 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.276861 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276725 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.277037 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.276874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45db0f5-c580-4237-b82d-4670b51665f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.277037 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.277007 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.277129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.277064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.277129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.277110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.279084 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.279058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45db0f5-c580-4237-b82d-4670b51665f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.279332 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.279314 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45db0f5-c580-4237-b82d-4670b51665f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.284480 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.284454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv46t\" (UniqueName: \"kubernetes.io/projected/e45db0f5-c580-4237-b82d-4670b51665f9-kube-api-access-nv46t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v9k2b\" (UID: \"e45db0f5-c580-4237-b82d-4670b51665f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.396775 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.396741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:20:53.531751 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:53.531707 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b"] Jul 16 12:20:53.532786 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:20:53.532761 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45db0f5_c580_4237_b82d_4670b51665f9.slice/crio-98d6aadd0fdffc021eab20966a0540039319a8fa2fbdfb87efb4be22cf36dc54 WatchSource:0}: Error finding container 98d6aadd0fdffc021eab20966a0540039319a8fa2fbdfb87efb4be22cf36dc54: Status 404 returned error can't find the container with id 98d6aadd0fdffc021eab20966a0540039319a8fa2fbdfb87efb4be22cf36dc54 Jul 16 12:20:54.178321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:54.178285 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" event={"ID":"e45db0f5-c580-4237-b82d-4670b51665f9","Type":"ContainerStarted","Data":"cc6219727affd647f134327d44e56f5f8348ac3b9e22d2693b185e373170e4fd"} Jul 16 12:20:54.178810 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:54.178345 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" event={"ID":"e45db0f5-c580-4237-b82d-4670b51665f9","Type":"ContainerStarted","Data":"98d6aadd0fdffc021eab20966a0540039319a8fa2fbdfb87efb4be22cf36dc54"} Jul 16 12:20:54.200970 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:20:54.200916 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" podStartSLOduration=0.93281273 podStartE2EDuration="1.200899266s" totalImagesPullingTime="268.086536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:20:54.198373023 +0000 UTC m=+851.892479755" watchObservedRunningTime="2026-07-16 12:20:54.200899266 +0000 UTC m=+851.895006011" Jul 16 12:21:03.397342 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:03.397299 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:21:03.397856 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:03.397370 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:21:03.409822 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:03.409788 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:21:04.235898 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:04.235871 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v9k2b" Jul 16 12:21:42.785400 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:42.785321 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:21:42.787030 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:21:42.786981 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:22:51.364302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.364262 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:22:51.364799 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.364489 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" podUID="90623247-c507-49e8-b5ab-522bbdf6408b" containerName="manager" containerID="cri-o://a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924" gracePeriod=30 Jul 16 12:22:51.619186 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.619127 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:22:51.726913 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.726875 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9cjf\" (UniqueName: \"kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf\") pod \"90623247-c507-49e8-b5ab-522bbdf6408b\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " Jul 16 12:22:51.727122 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.727025 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") pod \"90623247-c507-49e8-b5ab-522bbdf6408b\" (UID: \"90623247-c507-49e8-b5ab-522bbdf6408b\") " Jul 16 12:22:51.728973 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.728946 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "90623247-c507-49e8-b5ab-522bbdf6408b" (UID: "90623247-c507-49e8-b5ab-522bbdf6408b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:22:51.729112 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.729012 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf" (OuterVolumeSpecName: "kube-api-access-f9cjf") pod "90623247-c507-49e8-b5ab-522bbdf6408b" (UID: "90623247-c507-49e8-b5ab-522bbdf6408b"). InnerVolumeSpecName "kube-api-access-f9cjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:22:51.740445 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.740414 2583 generic.go:356] "Generic (PLEG): container finished" podID="90623247-c507-49e8-b5ab-522bbdf6408b" containerID="a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924" exitCode=0 Jul 16 12:22:51.740574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.740468 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" Jul 16 12:22:51.740574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.740491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" event={"ID":"90623247-c507-49e8-b5ab-522bbdf6408b","Type":"ContainerDied","Data":"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924"} Jul 16 12:22:51.740574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.740543 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lstfz" event={"ID":"90623247-c507-49e8-b5ab-522bbdf6408b","Type":"ContainerDied","Data":"9b813b17a98df94a4d9152514c209012c4b948a5af69e6f448df7818f5c5ff0e"} Jul 16 12:22:51.740574 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.740565 2583 scope.go:117] "RemoveContainer" containerID="a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924" Jul 16 12:22:51.749778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.749763 2583 scope.go:117] "RemoveContainer" containerID="a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924" Jul 16 12:22:51.750047 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:22:51.750021 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924\": container with ID starting with a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924 not found: ID does not exist" containerID="a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924" Jul 16 12:22:51.750148 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.750057 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924"} err="failed to get container status \"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924\": rpc error: code = NotFound desc = could not find container \"a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924\": container with ID starting with a839edcd280b380e1ee0cfb767e628067548d2fdc79accdf2279c2da84f7b924 not found: ID does not exist" Jul 16 12:22:51.764245 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.764217 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:22:51.770308 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.770283 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lstfz"] Jul 16 12:22:51.828232 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.828196 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90623247-c507-49e8-b5ab-522bbdf6408b-webhook-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:22:51.828232 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:51.828228 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9cjf\" (UniqueName: \"kubernetes.io/projected/90623247-c507-49e8-b5ab-522bbdf6408b-kube-api-access-f9cjf\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:22:52.794154 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:22:52.794121 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90623247-c507-49e8-b5ab-522bbdf6408b" path="/var/lib/kubelet/pods/90623247-c507-49e8-b5ab-522bbdf6408b/volumes" Jul 16 12:23:52.679890 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.679845 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:23:52.680515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.680405 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90623247-c507-49e8-b5ab-522bbdf6408b" containerName="manager" Jul 16 12:23:52.680515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.680427 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90623247-c507-49e8-b5ab-522bbdf6408b" containerName="manager" Jul 16 12:23:52.680684 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.680530 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90623247-c507-49e8-b5ab-522bbdf6408b" containerName="manager" Jul 16 12:23:52.684002 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.683962 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.687190 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.687170 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 12:23:52.688194 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.688175 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kkp9\"" Jul 16 12:23:52.688303 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.688250 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 12:23:52.700161 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.700135 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:23:52.771973 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.771937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.771973 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.771974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jbx\" (UniqueName: \"kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.872505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.872471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.872678 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.872521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jbx\" (UniqueName: \"kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.875042 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.875009 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.881905 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.881882 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jbx\" (UniqueName: \"kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx\") pod \"maas-controller-6dcdc7c8b8-lzrhl\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:52.994701 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:52.994573 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:53.153998 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:53.153959 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:23:53.155743 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:23:53.155715 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca78cbb_e9da_43f9_9411_5858e1fc771d.slice/crio-e3e6b8d0ebf19e70e23b316951f1dc88fe29746e9afa7d32363205dfec5487ec WatchSource:0}: Error finding container e3e6b8d0ebf19e70e23b316951f1dc88fe29746e9afa7d32363205dfec5487ec: Status 404 returned error can't find the container with id e3e6b8d0ebf19e70e23b316951f1dc88fe29746e9afa7d32363205dfec5487ec Jul 16 12:23:54.039250 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:54.039215 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" event={"ID":"cca78cbb-e9da-43f9-9411-5858e1fc771d","Type":"ContainerStarted","Data":"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af"} Jul 16 12:23:54.039684 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:54.039270 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" event={"ID":"cca78cbb-e9da-43f9-9411-5858e1fc771d","Type":"ContainerStarted","Data":"e3e6b8d0ebf19e70e23b316951f1dc88fe29746e9afa7d32363205dfec5487ec"} Jul 16 12:23:54.039684 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:54.039290 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:23:54.060150 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:23:54.060101 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" podStartSLOduration=1.353042157 podStartE2EDuration="2.060086097s" totalImagesPullingTime="707.04394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:23:54.05886601 +0000 UTC m=+1031.752972756" watchObservedRunningTime="2026-07-16 12:23:54.060086097 +0000 UTC m=+1031.754192815" Jul 16 12:24:05.046926 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:05.046893 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:24:09.771455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:09.771422 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:24:09.771847 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:09.771625 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" podUID="cca78cbb-e9da-43f9-9411-5858e1fc771d" containerName="manager" containerID="cri-o://57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af" gracePeriod=30 Jul 16 12:24:10.019683 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.019660 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:24:10.113897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.113796 2583 generic.go:356] "Generic (PLEG): container finished" podID="cca78cbb-e9da-43f9-9411-5858e1fc771d" containerID="57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af" exitCode=0 Jul 16 12:24:10.113897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.113848 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" Jul 16 12:24:10.113897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.113882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" event={"ID":"cca78cbb-e9da-43f9-9411-5858e1fc771d","Type":"ContainerDied","Data":"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af"} Jul 16 12:24:10.114183 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.113944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-lzrhl" event={"ID":"cca78cbb-e9da-43f9-9411-5858e1fc771d","Type":"ContainerDied","Data":"e3e6b8d0ebf19e70e23b316951f1dc88fe29746e9afa7d32363205dfec5487ec"} Jul 16 12:24:10.114183 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.113968 2583 scope.go:117] "RemoveContainer" containerID="57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af" Jul 16 12:24:10.122221 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.122205 2583 scope.go:117] "RemoveContainer" containerID="57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af" Jul 16 12:24:10.122473 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:24:10.122452 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af\": container with ID starting with 57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af not found: ID does not exist" containerID="57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af" Jul 16 12:24:10.122537 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.122487 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af"} err="failed to get container status \"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af\": rpc error: code = NotFound desc = could not find container \"57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af\": container with ID starting with 57b996c8ebc4d86766c4e715561e28630a5832cca66fedcf8feeba3eb90e97af not found: ID does not exist" Jul 16 12:24:10.124778 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.124761 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert\") pod \"cca78cbb-e9da-43f9-9411-5858e1fc771d\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " Jul 16 12:24:10.124833 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.124816 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4jbx\" (UniqueName: \"kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx\") pod \"cca78cbb-e9da-43f9-9411-5858e1fc771d\" (UID: \"cca78cbb-e9da-43f9-9411-5858e1fc771d\") " Jul 16 12:24:10.126779 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.126758 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx" (OuterVolumeSpecName: "kube-api-access-h4jbx") pod "cca78cbb-e9da-43f9-9411-5858e1fc771d" (UID: "cca78cbb-e9da-43f9-9411-5858e1fc771d"). InnerVolumeSpecName "kube-api-access-h4jbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:24:10.126875 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.126812 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cca78cbb-e9da-43f9-9411-5858e1fc771d" (UID: "cca78cbb-e9da-43f9-9411-5858e1fc771d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:24:10.225725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.225681 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jbx\" (UniqueName: \"kubernetes.io/projected/cca78cbb-e9da-43f9-9411-5858e1fc771d-kube-api-access-h4jbx\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:24:10.225725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.225719 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cca78cbb-e9da-43f9-9411-5858e1fc771d-webhook-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:24:10.438235 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.438198 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:24:10.443195 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.443165 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-lzrhl"] Jul 16 12:24:10.794410 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:10.794315 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca78cbb-e9da-43f9-9411-5858e1fc771d" path="/var/lib/kubelet/pods/cca78cbb-e9da-43f9-9411-5858e1fc771d/volumes" Jul 16 12:24:11.336091 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.336047 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-7c9zr"] Jul 16 12:24:11.336486 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.336461 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca78cbb-e9da-43f9-9411-5858e1fc771d" containerName="manager" Jul 16 12:24:11.336541 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.336489 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca78cbb-e9da-43f9-9411-5858e1fc771d" containerName="manager" Jul 16 12:24:11.336575 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.336550 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca78cbb-e9da-43f9-9411-5858e1fc771d" containerName="manager" Jul 16 12:24:11.339449 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.339432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.342144 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.342121 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 12:24:11.342291 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.342155 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 12:24:11.343239 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.343212 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kkp9\"" Jul 16 12:24:11.348834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.348808 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-7c9zr"] Jul 16 12:24:11.436785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.436743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjdh\" (UniqueName: \"kubernetes.io/projected/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-kube-api-access-mmjdh\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.436970 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.436822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.537981 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.537927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjdh\" (UniqueName: \"kubernetes.io/projected/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-kube-api-access-mmjdh\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.538225 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.538047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.540444 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.540423 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-webhook-cert\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.549173 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.549142 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjdh\" (UniqueName: \"kubernetes.io/projected/7c86bdd7-9ba4-46fa-8684-831706f7a3f2-kube-api-access-mmjdh\") pod \"maas-controller-6dcdc7c8b8-7c9zr\" (UID: \"7c86bdd7-9ba4-46fa-8684-831706f7a3f2\") " pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.650863 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.650823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:11.788766 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:11.788741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dcdc7c8b8-7c9zr"] Jul 16 12:24:11.790731 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:24:11.790703 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c86bdd7_9ba4_46fa_8684_831706f7a3f2.slice/crio-b15b41389bfd130e9fbd94177a24e979e0a41b6a75c1485ca5531cfe5231fa6c WatchSource:0}: Error finding container b15b41389bfd130e9fbd94177a24e979e0a41b6a75c1485ca5531cfe5231fa6c: Status 404 returned error can't find the container with id b15b41389bfd130e9fbd94177a24e979e0a41b6a75c1485ca5531cfe5231fa6c Jul 16 12:24:12.125561 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:12.125529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" event={"ID":"7c86bdd7-9ba4-46fa-8684-831706f7a3f2","Type":"ContainerStarted","Data":"b15b41389bfd130e9fbd94177a24e979e0a41b6a75c1485ca5531cfe5231fa6c"} Jul 16 12:24:13.130761 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:13.130723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" event={"ID":"7c86bdd7-9ba4-46fa-8684-831706f7a3f2","Type":"ContainerStarted","Data":"abefc66856030c327805f385b17903125651a7a3471fc83a1fa87dc5e448bfbd"} Jul 16 12:24:13.131166 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:13.130836 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:24:13.150931 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:13.150882 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" podStartSLOduration=1.766172331 podStartE2EDuration="2.15086609s" totalImagesPullingTime="384.693759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:24:13.149427525 +0000 UTC m=+1050.843534245" watchObservedRunningTime="2026-07-16 12:24:13.15086609 +0000 UTC m=+1050.844972824" Jul 16 12:24:24.139798 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:24:24.139763 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dcdc7c8b8-7c9zr" Jul 16 12:26:42.813258 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:26:42.813228 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:26:42.816056 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:26:42.816029 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:31:42.844214 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:31:42.844180 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:31:42.849228 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:31:42.849202 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:36:20.071795 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.071757 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:20.075181 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.075155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.078135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.078114 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c7727fcf-openshift-default-dockercfg-ngsm8\"" Jul 16 12:36:20.112215 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.112175 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:20.113887 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.113862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114050 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.113910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114050 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.113947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114050 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d4k\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114217 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114260 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.114380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.114309 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.215817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d4k\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.215971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216053 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216393 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216393 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216518 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216518 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216484 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216712 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.216878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.216866 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.218205 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.218184 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.218532 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.218514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.237436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.237394 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.237639 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.237533 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d4k\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k\") pod \"e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.386491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.386452 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:20.547386 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.547359 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:20.549533 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:36:20.549503 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741a996e_e41a_46c2_ab80_f67fb9f56701.slice/crio-543e0e49a80670eb58dc67520dbd9315b4ef10eb95a77daded7cda2456259190 WatchSource:0}: Error finding container 543e0e49a80670eb58dc67520dbd9315b4ef10eb95a77daded7cda2456259190: Status 404 returned error can't find the container with id 543e0e49a80670eb58dc67520dbd9315b4ef10eb95a77daded7cda2456259190 Jul 16 12:36:20.551879 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.551849 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:20.551955 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.551918 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:20.551955 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.551948 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:20.558202 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:20.558175 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" event={"ID":"741a996e-e41a-46c2-ab80-f67fb9f56701","Type":"ContainerStarted","Data":"543e0e49a80670eb58dc67520dbd9315b4ef10eb95a77daded7cda2456259190"} Jul 16 12:36:21.565318 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:21.565284 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" event={"ID":"741a996e-e41a-46c2-ab80-f67fb9f56701","Type":"ContainerStarted","Data":"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9"} Jul 16 12:36:22.387356 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:22.387315 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:22.392496 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:22.392463 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:22.419120 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:22.419057 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" podStartSLOduration=2.419039277 podStartE2EDuration="2.419039277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:21.59342834 +0000 UTC m=+1779.287535083" watchObservedRunningTime="2026-07-16 12:36:22.419039277 +0000 UTC m=+1780.113145995" Jul 16 12:36:22.571546 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:22.571514 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:22.572655 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:22.572637 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:26.617681 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:26.617639 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:26.618742 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:26.617841 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" podUID="741a996e-e41a-46c2-ab80-f67fb9f56701" containerName="istio-proxy" containerID="cri-o://6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9" gracePeriod=30 Jul 16 12:36:32.271873 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.271844 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:32.433461 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433418 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433461 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433462 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433521 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433549 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433578 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433605 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433666 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.433739 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433717 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.434142 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433746 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9d4k\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k\") pod \"741a996e-e41a-46c2-ab80-f67fb9f56701\" (UID: \"741a996e-e41a-46c2-ab80-f67fb9f56701\") " Jul 16 12:36:32.434142 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.433746 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:32.434142 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.434044 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-credential-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.434142 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.434086 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:32.434142 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.434111 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:36:32.434435 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.434409 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:32.434736 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.434705 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data" (OuterVolumeSpecName: "istio-data") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:32.435967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.435934 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:32.435967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.435952 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k" (OuterVolumeSpecName: "kube-api-access-p9d4k") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "kube-api-access-p9d4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:32.436257 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.436235 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 12:36:32.436317 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.436246 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token" (OuterVolumeSpecName: "istio-token") pod "741a996e-e41a-46c2-ab80-f67fb9f56701" (UID: "741a996e-e41a-46c2-ab80-f67fb9f56701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:32.534698 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534657 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741a996e-e41a-46c2-ab80-f67fb9f56701-istiod-ca-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534698 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534692 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-envoy\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534698 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534700 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534698 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534708 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-podinfo\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534717 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534725 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-istio-data\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534733 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d4k\" (UniqueName: \"kubernetes.io/projected/741a996e-e41a-46c2-ab80-f67fb9f56701-kube-api-access-p9d4k\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.534971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.534742 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741a996e-e41a-46c2-ab80-f67fb9f56701-workload-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:32.623046 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.623011 2583 generic.go:356] "Generic (PLEG): container finished" podID="741a996e-e41a-46c2-ab80-f67fb9f56701" containerID="6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9" exitCode=0 Jul 16 12:36:32.623265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.623051 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" Jul 16 12:36:32.623265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.623096 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" event={"ID":"741a996e-e41a-46c2-ab80-f67fb9f56701","Type":"ContainerDied","Data":"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9"} Jul 16 12:36:32.623265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.623154 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9" event={"ID":"741a996e-e41a-46c2-ab80-f67fb9f56701","Type":"ContainerDied","Data":"543e0e49a80670eb58dc67520dbd9315b4ef10eb95a77daded7cda2456259190"} Jul 16 12:36:32.623265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.623195 2583 scope.go:117] "RemoveContainer" containerID="6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9" Jul 16 12:36:32.636600 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.636577 2583 scope.go:117] "RemoveContainer" containerID="6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9" Jul 16 12:36:32.636964 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:36:32.636940 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9\": container with ID starting with 6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9 not found: ID does not exist" containerID="6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9" Jul 16 12:36:32.637124 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.636977 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9"} err="failed to get container status \"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9\": rpc error: code = NotFound desc = could not find container \"6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9\": container with ID starting with 6b38291e2987842ef60773c830847f0d91d47a1db0f23328d244c477e1ce52e9 not found: ID does not exist" Jul 16 12:36:32.646380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.646352 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:32.656378 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.656348 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c7727fcf-openshift-default-cd8fdf6b9-cngp9"] Jul 16 12:36:32.794382 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:32.794302 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741a996e-e41a-46c2-ab80-f67fb9f56701" path="/var/lib/kubelet/pods/741a996e-e41a-46c2-ab80-f67fb9f56701/volumes" Jul 16 12:36:33.561434 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.561390 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:33.565646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.562306 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741a996e-e41a-46c2-ab80-f67fb9f56701" containerName="istio-proxy" Jul 16 12:36:33.565646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.562332 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="741a996e-e41a-46c2-ab80-f67fb9f56701" containerName="istio-proxy" Jul 16 12:36:33.565646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.562562 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="741a996e-e41a-46c2-ab80-f67fb9f56701" containerName="istio-proxy" Jul 16 12:36:33.566767 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.566737 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.571529 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.571508 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-051a7b7b-gw-openshift-default-dockercfg-8jnld\"" Jul 16 12:36:33.583362 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.583328 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:33.645644 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645607 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.645848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645663 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.645848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.645848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2f9\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.645848 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.646128 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.646128 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.645975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.646128 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.646054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.646254 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.646152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.746880 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.746842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.746889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.746910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.746952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.746974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2f9\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747360 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747624 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747424 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747624 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747494 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.747974 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.747949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.749494 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.749470 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.749725 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.749707 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.755376 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.755352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.755480 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.755436 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2f9\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9\") pod \"e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:33.881450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:33.881410 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:34.044865 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.044834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:34.047592 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:36:34.047561 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf825596_5e5c_4e4d_9b9c_3e2d3859fa85.slice/crio-3131cbcd00c0217275ebbf8d4747e728864f7bc1de5e67c4686851bd15f9609d WatchSource:0}: Error finding container 3131cbcd00c0217275ebbf8d4747e728864f7bc1de5e67c4686851bd15f9609d: Status 404 returned error can't find the container with id 3131cbcd00c0217275ebbf8d4747e728864f7bc1de5e67c4686851bd15f9609d Jul 16 12:36:34.049951 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.049910 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:34.050063 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.050023 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:34.050105 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.050070 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:34.637458 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.637415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" event={"ID":"af825596-5e5c-4e4d-9b9c-3e2d3859fa85","Type":"ContainerStarted","Data":"f589887172389116db3fade83105fef3eb632fdd4ce720d2e19dc750f45d9c62"} Jul 16 12:36:34.637860 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.637481 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" event={"ID":"af825596-5e5c-4e4d-9b9c-3e2d3859fa85","Type":"ContainerStarted","Data":"3131cbcd00c0217275ebbf8d4747e728864f7bc1de5e67c4686851bd15f9609d"} Jul 16 12:36:34.662014 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.661935 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" podStartSLOduration=1.661916242 podStartE2EDuration="1.661916242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:34.659741023 +0000 UTC m=+1792.353847742" watchObservedRunningTime="2026-07-16 12:36:34.661916242 +0000 UTC m=+1792.356022960" Jul 16 12:36:34.882560 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:34.882520 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:35.887297 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:35.887258 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:36.647968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:36.647933 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:36.648965 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:36.648946 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:39.553510 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:39.553465 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:39.661872 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:39.661822 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" podUID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" containerName="istio-proxy" containerID="cri-o://f589887172389116db3fade83105fef3eb632fdd4ce720d2e19dc750f45d9c62" gracePeriod=30 Jul 16 12:36:42.874032 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:42.873999 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:36:42.879668 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:42.879645 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:36:45.698689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.698655 2583 generic.go:356] "Generic (PLEG): container finished" podID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" containerID="f589887172389116db3fade83105fef3eb632fdd4ce720d2e19dc750f45d9c62" exitCode=0 Jul 16 12:36:45.698689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.698700 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" event={"ID":"af825596-5e5c-4e4d-9b9c-3e2d3859fa85","Type":"ContainerDied","Data":"f589887172389116db3fade83105fef3eb632fdd4ce720d2e19dc750f45d9c62"} Jul 16 12:36:45.729153 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.729085 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:45.860212 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860173 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860260 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860279 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860300 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860320 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860355 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860409 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860380 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860416 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860458 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt2f9\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9\") pod \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\" (UID: \"af825596-5e5c-4e4d-9b9c-3e2d3859fa85\") " Jul 16 12:36:45.860745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860727 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:45.860890 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860815 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:36:45.860890 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.860834 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.861070 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.861044 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:45.861286 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.861257 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data" (OuterVolumeSpecName: "istio-data") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:45.861286 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.861111 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:45.862613 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.862579 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 12:36:45.862718 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.862678 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:45.862856 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.862841 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token" (OuterVolumeSpecName: "istio-token") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:45.862913 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.862903 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9" (OuterVolumeSpecName: "kube-api-access-kt2f9") pod "af825596-5e5c-4e4d-9b9c-3e2d3859fa85" (UID: "af825596-5e5c-4e4d-9b9c-3e2d3859fa85"). InnerVolumeSpecName "kube-api-access-kt2f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:45.961544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961497 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961536 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-data\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961544 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961551 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istiod-ca-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961560 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-credential-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961569 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-envoy\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961578 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-istio-podinfo\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961585 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-workload-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:45.961835 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:45.961593 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2f9\" (UniqueName: \"kubernetes.io/projected/af825596-5e5c-4e4d-9b9c-3e2d3859fa85-kube-api-access-kt2f9\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:46.589329 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.589297 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:36:46.589677 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.589665 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" containerName="istio-proxy" Jul 16 12:36:46.589727 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.589678 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" containerName="istio-proxy" Jul 16 12:36:46.589758 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.589748 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" containerName="istio-proxy" Jul 16 12:36:46.591893 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.591877 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.595504 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.595478 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-15231c4d-openshift-default-dockercfg-6d572\"" Jul 16 12:36:46.614043 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.614011 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:36:46.706665 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.706634 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" Jul 16 12:36:46.707177 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.706638 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44" event={"ID":"af825596-5e5c-4e4d-9b9c-3e2d3859fa85","Type":"ContainerDied","Data":"3131cbcd00c0217275ebbf8d4747e728864f7bc1de5e67c4686851bd15f9609d"} Jul 16 12:36:46.707177 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.706803 2583 scope.go:117] "RemoveContainer" containerID="f589887172389116db3fade83105fef3eb632fdd4ce720d2e19dc750f45d9c62" Jul 16 12:36:46.753529 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.753487 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:46.760827 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.760796 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-051a7b7b-gw-openshift-default-854959d54c-2rt44"] Jul 16 12:36:46.770424 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.770595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.770595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.770595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.771017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.771017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.771017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.771017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.771017 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.770746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5rz\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.795626 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.795580 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af825596-5e5c-4e4d-9b9c-3e2d3859fa85" path="/var/lib/kubelet/pods/af825596-5e5c-4e4d-9b9c-3e2d3859fa85/volumes" Jul 16 12:36:46.872060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.872288 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.872288 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.872288 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.872288 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5rz\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.872288 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.872800 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj"] Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.873029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.873157 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.873171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.873506 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.873587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.873518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.874837 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.874812 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.875029 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.875008 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.876571 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.876555 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:46.879965 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.879948 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-15231c4d-serving-cert\"" Jul 16 12:36:46.890548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.890474 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj"] Jul 16 12:36:46.892038 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.891979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5rz\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.892244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.892217 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token\") pod \"e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.903219 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.903190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:46.973417 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.973368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a84a17-805d-4cfe-be35-b32a43ecfcf8-openshift-service-ca\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:46.973569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.973467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:46.973569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:46.973544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzbm\" (UniqueName: \"kubernetes.io/projected/41a84a17-805d-4cfe-be35-b32a43ecfcf8-kube-api-access-8zzbm\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.074452 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.074412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.074659 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.074484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzbm\" (UniqueName: \"kubernetes.io/projected/41a84a17-805d-4cfe-be35-b32a43ecfcf8-kube-api-access-8zzbm\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.074659 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.074549 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a84a17-805d-4cfe-be35-b32a43ecfcf8-openshift-service-ca\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.074659 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:36:47.074559 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-serving-cert: secret "maas-api-e2e-derive-15231c4d-serving-cert" not found Jul 16 12:36:47.074659 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:36:47.074627 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls podName:41a84a17-805d-4cfe-be35-b32a43ecfcf8 nodeName:}" failed. No retries permitted until 2026-07-16 12:36:47.57460988 +0000 UTC m=+1805.268716598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls") pod "maas-api-e2e-derive-15231c4d-67677895c4-p55xj" (UID: "41a84a17-805d-4cfe-be35-b32a43ecfcf8") : secret "maas-api-e2e-derive-15231c4d-serving-cert" not found Jul 16 12:36:47.075226 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.075205 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a84a17-805d-4cfe-be35-b32a43ecfcf8-openshift-service-ca\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.099646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.099608 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzbm\" (UniqueName: \"kubernetes.io/projected/41a84a17-805d-4cfe-be35-b32a43ecfcf8-kube-api-access-8zzbm\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.112359 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.112299 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:36:47.114643 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:36:47.114615 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2ab1cb_20e8_4d9f_829d_cf96327982e4.slice/crio-91e1ac9291651f39fd3054bf3ec2dda87f5a46dfcb9db3138d03f75039d380ea WatchSource:0}: Error finding container 91e1ac9291651f39fd3054bf3ec2dda87f5a46dfcb9db3138d03f75039d380ea: Status 404 returned error can't find the container with id 91e1ac9291651f39fd3054bf3ec2dda87f5a46dfcb9db3138d03f75039d380ea Jul 16 12:36:47.116905 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.116876 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:47.116967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.116945 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:47.117025 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.116978 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:36:47.580058 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.579922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.582325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.582296 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a84a17-805d-4cfe-be35-b32a43ecfcf8-maas-api-tls\") pod \"maas-api-e2e-derive-15231c4d-67677895c4-p55xj\" (UID: \"41a84a17-805d-4cfe-be35-b32a43ecfcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.714052 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.713980 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" event={"ID":"eb2ab1cb-20e8-4d9f-829d-cf96327982e4","Type":"ContainerStarted","Data":"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce"} Jul 16 12:36:47.714429 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.714102 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" event={"ID":"eb2ab1cb-20e8-4d9f-829d-cf96327982e4","Type":"ContainerStarted","Data":"91e1ac9291651f39fd3054bf3ec2dda87f5a46dfcb9db3138d03f75039d380ea"} Jul 16 12:36:47.809066 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.809016 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" podStartSLOduration=1.809001244 podStartE2EDuration="1.809001244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:47.807669683 +0000 UTC m=+1805.501776401" watchObservedRunningTime="2026-07-16 12:36:47.809001244 +0000 UTC m=+1805.503107955" Jul 16 12:36:47.810000 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.809961 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:47.906155 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.903329 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:47.991722 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.991542 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj"] Jul 16 12:36:47.994135 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:36:47.994109 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a84a17_805d_4cfe_be35_b32a43ecfcf8.slice/crio-8008da1ba007e1ea16dcb9685a63ffad2d17cc4e4c5dc9f03f0b7e31b7b629e0 WatchSource:0}: Error finding container 8008da1ba007e1ea16dcb9685a63ffad2d17cc4e4c5dc9f03f0b7e31b7b629e0: Status 404 returned error can't find the container with id 8008da1ba007e1ea16dcb9685a63ffad2d17cc4e4c5dc9f03f0b7e31b7b629e0 Jul 16 12:36:47.996165 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:47.996145 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:36:48.721438 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:48.721331 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" event={"ID":"41a84a17-805d-4cfe-be35-b32a43ecfcf8","Type":"ContainerStarted","Data":"a46d9275a8d3b8a8bc20ce95b914cc095233109b8791ce87cedd69ce0e1ee404"} Jul 16 12:36:48.721438 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:48.721404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" event={"ID":"41a84a17-805d-4cfe-be35-b32a43ecfcf8","Type":"ContainerStarted","Data":"8008da1ba007e1ea16dcb9685a63ffad2d17cc4e4c5dc9f03f0b7e31b7b629e0"} Jul 16 12:36:48.721889 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:48.721617 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:48.799527 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:48.799474 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" podStartSLOduration=2.402854556 podStartE2EDuration="2.799456842s" totalImagesPullingTime="396.602286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:36:48.799310249 +0000 UTC m=+1806.493416970" watchObservedRunningTime="2026-07-16 12:36:48.799456842 +0000 UTC m=+1806.493563561" Jul 16 12:36:48.909527 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:48.909497 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:49.727661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:49.727622 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:49.728467 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:49.728444 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:52.365896 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:52.365860 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:36:52.753385 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:52.753335 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" podUID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" containerName="istio-proxy" containerID="cri-o://a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce" gracePeriod=30 Jul 16 12:36:54.740446 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:54.740375 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15231c4d-67677895c4-p55xj" Jul 16 12:36:58.608089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.608058 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:58.695390 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695390 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695390 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695363 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695390 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695395 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695433 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695448 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n5rz\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695467 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695492 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695548 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo\") pod \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\" (UID: \"eb2ab1cb-20e8-4d9f-829d-cf96327982e4\") " Jul 16 12:36:58.695738 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695626 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:58.696060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695739 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:58.696060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695909 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-credential-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.696060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.695929 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.696060 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.696022 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:58.696273 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.696193 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:36:58.696273 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.696223 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:58.697878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.697845 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz" (OuterVolumeSpecName: "kube-api-access-6n5rz") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "kube-api-access-6n5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:58.697878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.697849 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 12:36:58.698126 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.698108 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:36:58.698172 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.698120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb2ab1cb-20e8-4d9f-829d-cf96327982e4" (UID: "eb2ab1cb-20e8-4d9f-829d-cf96327982e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:36:58.787505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.787468 2583 generic.go:356] "Generic (PLEG): container finished" podID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" containerID="a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce" exitCode=0 Jul 16 12:36:58.787708 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.787519 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" Jul 16 12:36:58.787708 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.787549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" event={"ID":"eb2ab1cb-20e8-4d9f-829d-cf96327982e4","Type":"ContainerDied","Data":"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce"} Jul 16 12:36:58.787708 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.787668 2583 scope.go:117] "RemoveContainer" containerID="a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce" Jul 16 12:36:58.795429 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.795401 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk" event={"ID":"eb2ab1cb-20e8-4d9f-829d-cf96327982e4","Type":"ContainerDied","Data":"91e1ac9291651f39fd3054bf3ec2dda87f5a46dfcb9db3138d03f75039d380ea"} Jul 16 12:36:58.796651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796619 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-data\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796651 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796653 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-envoy\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796666 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796680 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5rz\" (UniqueName: \"kubernetes.io/projected/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-kube-api-access-6n5rz\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796692 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istiod-ca-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796702 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-workload-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.796818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.796710 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb2ab1cb-20e8-4d9f-829d-cf96327982e4-istio-podinfo\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:36:58.797950 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.797933 2583 scope.go:117] "RemoveContainer" containerID="a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce" Jul 16 12:36:58.798233 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:36:58.798210 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce\": container with ID starting with a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce not found: ID does not exist" containerID="a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce" Jul 16 12:36:58.798302 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.798239 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce"} err="failed to get container status \"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce\": rpc error: code = NotFound desc = could not find container \"a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce\": container with ID starting with a5f1924ac3549c2dab018c16d6a610e2ece4d1058e9ef9c24b93ae41b7144fce not found: ID does not exist" Jul 16 12:36:58.819162 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.819125 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:36:58.820918 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:36:58.820899 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-15231c4d-openshift-default-6dcf866955-jzddk"] Jul 16 12:37:00.794132 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:37:00.794087 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" path="/var/lib/kubelet/pods/eb2ab1cb-20e8-4d9f-829d-cf96327982e4/volumes" Jul 16 12:38:41.108878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.108828 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:38:41.109381 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.109367 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" containerName="istio-proxy" Jul 16 12:38:41.109426 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.109385 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" containerName="istio-proxy" Jul 16 12:38:41.109465 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.109445 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2ab1cb-20e8-4d9f-829d-cf96327982e4" containerName="istio-proxy" Jul 16 12:38:41.111863 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.111843 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.115760 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.115727 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df8d3cf1-openshift-default-dockercfg-5tmmw\"" Jul 16 12:38:41.141851 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.141811 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:38:41.249086 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249291 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249291 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249291 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249291 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249241 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jflq\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.249436 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.249419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350453 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350453 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jflq\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350530 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.350923 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350913 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.351128 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.350934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.351128 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.351100 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.351229 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.351180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.351380 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.351361 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.351440 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.351376 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.353077 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.353054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.353216 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.353164 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.363149 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.363073 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.363488 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.363467 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jflq\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq\") pod \"e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.425311 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.425268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:41.580790 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.580758 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:38:41.583797 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:38:41.583760 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e92818_54eb_40af_89f1_09e2b50c823f.slice/crio-10e903381aaee72078b7aae7abe7776bfc1fb9ece08daab021008db31a8a9bea WatchSource:0}: Error finding container 10e903381aaee72078b7aae7abe7776bfc1fb9ece08daab021008db31a8a9bea: Status 404 returned error can't find the container with id 10e903381aaee72078b7aae7abe7776bfc1fb9ece08daab021008db31a8a9bea Jul 16 12:38:41.586253 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.586216 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:41.586346 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.586294 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:41.586346 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:41.586325 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:42.288742 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:42.288695 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerStarted","Data":"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9"} Jul 16 12:38:42.289280 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:42.288758 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerStarted","Data":"10e903381aaee72078b7aae7abe7776bfc1fb9ece08daab021008db31a8a9bea"} Jul 16 12:38:42.313353 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:42.313292 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podStartSLOduration=1.313269685 podStartE2EDuration="1.313269685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:38:42.311895174 +0000 UTC m=+1920.006001895" watchObservedRunningTime="2026-07-16 12:38:42.313269685 +0000 UTC m=+1920.007376403" Jul 16 12:38:42.425606 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:42.425488 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:42.433588 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:42.433549 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:43.294326 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:43.294289 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:43.295583 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:43.295554 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:56.366351 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.366312 2583 generic.go:356] "Generic (PLEG): container finished" podID="08e92818-54eb-40af-89f1-09e2b50c823f" containerID="c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9" exitCode=0 Jul 16 12:38:56.366857 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.366383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerDied","Data":"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9"} Jul 16 12:38:56.366857 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.366811 2583 scope.go:117] "RemoveContainer" containerID="c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9" Jul 16 12:38:56.367891 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.367862 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:56.367967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.367950 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:56.368030 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:56.368018 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:38:57.373489 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:57.373452 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerStarted","Data":"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1"} Jul 16 12:38:57.425733 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:57.425691 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:57.426931 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:57.426897 2583 patch_prober.go:28] interesting pod/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 12:38:57.427097 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:57.426958 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 12:38:58.294135 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:58.294087 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:38:58.426270 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:58.426220 2583 patch_prober.go:28] interesting pod/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 12:38:58.426782 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:58.426303 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 12:38:59.426743 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:59.426701 2583 patch_prober.go:28] interesting pod/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 12:38:59.427376 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:38:59.427339 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 12:39:00.426037 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:00.425952 2583 patch_prober.go:28] interesting pod/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 12:39:00.426319 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:00.426082 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 12:39:01.426213 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:01.426167 2583 patch_prober.go:28] interesting pod/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 12:39:01.426971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:01.426932 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 12:39:02.430243 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:02.430210 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:39:02.431552 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:02.431530 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:39:04.186812 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:04.186770 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:39:04.407656 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:04.407607 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" containerID="cri-o://59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1" gracePeriod=30 Jul 16 12:39:09.675564 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.675482 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:39:09.729203 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729166 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729249 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jflq\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729374 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729422 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729396 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729457 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729517 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729516 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:39:09.729680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729543 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert\") pod \"08e92818-54eb-40af-89f1-09e2b50c823f\" (UID: \"08e92818-54eb-40af-89f1-09e2b50c823f\") " Jul 16 12:39:09.729885 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.729808 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.730156 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.730126 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:39:09.730309 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.730265 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data" (OuterVolumeSpecName: "istio-data") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:39:09.730500 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.730462 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:39:09.730830 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.730806 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:39:09.732633 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.732604 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 12:39:09.732833 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.732807 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:39:09.732895 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.732845 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token" (OuterVolumeSpecName: "istio-token") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:39:09.733439 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.733421 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq" (OuterVolumeSpecName: "kube-api-access-9jflq") pod "08e92818-54eb-40af-89f1-09e2b50c823f" (UID: "08e92818-54eb-40af-89f1-09e2b50c823f"). InnerVolumeSpecName "kube-api-access-9jflq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:39:09.831256 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831210 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-envoy\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831256 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831248 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jflq\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-kube-api-access-9jflq\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831256 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831260 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-istio-data\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831256 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831268 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e92818-54eb-40af-89f1-09e2b50c823f-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831276 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-workload-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831284 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e92818-54eb-40af-89f1-09e2b50c823f-credential-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831292 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e92818-54eb-40af-89f1-09e2b50c823f-istio-podinfo\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:09.831548 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:09.831300 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e92818-54eb-40af-89f1-09e2b50c823f-istiod-ca-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:39:10.444463 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.444411 2583 generic.go:356] "Generic (PLEG): container finished" podID="08e92818-54eb-40af-89f1-09e2b50c823f" containerID="59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1" exitCode=0 Jul 16 12:39:10.444670 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.444483 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" Jul 16 12:39:10.444670 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.444500 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerDied","Data":"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1"} Jul 16 12:39:10.444670 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.444554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr" event={"ID":"08e92818-54eb-40af-89f1-09e2b50c823f","Type":"ContainerDied","Data":"10e903381aaee72078b7aae7abe7776bfc1fb9ece08daab021008db31a8a9bea"} Jul 16 12:39:10.444670 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.444578 2583 scope.go:117] "RemoveContainer" containerID="59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1" Jul 16 12:39:10.454822 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.454797 2583 scope.go:117] "RemoveContainer" containerID="c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9" Jul 16 12:39:10.464262 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.464236 2583 scope.go:117] "RemoveContainer" containerID="59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1" Jul 16 12:39:10.464601 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:39:10.464577 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1\": container with ID starting with 59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1 not found: ID does not exist" containerID="59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1" Jul 16 12:39:10.464713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.464614 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1"} err="failed to get container status \"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1\": rpc error: code = NotFound desc = could not find container \"59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1\": container with ID starting with 59bffe5fbb513d8ec4f7678a665901a35ec4334641002ad526e0523ee944ebb1 not found: ID does not exist" Jul 16 12:39:10.464713 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.464634 2583 scope.go:117] "RemoveContainer" containerID="c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9" Jul 16 12:39:10.464934 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:39:10.464915 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9\": container with ID starting with c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9 not found: ID does not exist" containerID="c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9" Jul 16 12:39:10.464980 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.464941 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9"} err="failed to get container status \"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9\": rpc error: code = NotFound desc = could not find container \"c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9\": container with ID starting with c375b2ccaec4b6cbec5fcb4bcaa90bde6ae63ed5c9bdc0534aed2a82bcf94fb9 not found: ID does not exist" Jul 16 12:39:10.468063 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.468033 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:39:10.473203 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.473174 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df8d3cf1-openshift-default-6bb8b4b54c-sv7qr"] Jul 16 12:39:10.794468 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:39:10.794380 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" path="/var/lib/kubelet/pods/08e92818-54eb-40af-89f1-09e2b50c823f/volumes" Jul 16 12:40:32.663917 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.663879 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664266 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664278 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664297 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664303 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664356 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.664370 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.664363 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e92818-54eb-40af-89f1-09e2b50c823f" containerName="istio-proxy" Jul 16 12:40:32.666428 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.666405 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.669609 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.669586 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9ac0a7-openshift-default-dockercfg-npmwq\"" Jul 16 12:40:32.685052 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.684976 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:40:32.693003 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.692944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693200 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693200 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693200 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttlw\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693321 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693305 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.693452 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.693326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794522 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794689 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794667 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794875 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.794875 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.794721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.795369 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.795340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.795587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.795468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.795587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.795506 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.795767 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.795593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttlw\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.797929 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.795966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.797929 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.796064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.797929 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.796322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.802920 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.802891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.803338 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.803318 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.806834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.806801 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttlw\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.811275 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.811249 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token\") pod \"e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:32.978251 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:32.978145 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:33.147969 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.147940 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:40:33.150739 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:40:33.150707 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c3b59_429d_46e1_85bb_77d6b9d837b7.slice/crio-da737630b7b1cbd0f7a1d7864ae086fb08120b13c60f2a2a209fdd4e90947b42 WatchSource:0}: Error finding container da737630b7b1cbd0f7a1d7864ae086fb08120b13c60f2a2a209fdd4e90947b42: Status 404 returned error can't find the container with id da737630b7b1cbd0f7a1d7864ae086fb08120b13c60f2a2a209fdd4e90947b42 Jul 16 12:40:33.152864 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.152826 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:33.152959 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.152898 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:33.152959 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.152929 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:33.858056 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.858016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerStarted","Data":"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987"} Jul 16 12:40:33.858566 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.858082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerStarted","Data":"da737630b7b1cbd0f7a1d7864ae086fb08120b13c60f2a2a209fdd4e90947b42"} Jul 16 12:40:33.883537 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.883467 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podStartSLOduration=1.883444392 podStartE2EDuration="1.883444392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:33.883225222 +0000 UTC m=+2031.577331962" watchObservedRunningTime="2026-07-16 12:40:33.883444392 +0000 UTC m=+2031.577551112" Jul 16 12:40:33.978505 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:33.978464 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:34.984939 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:34.984902 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:35.867635 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:35.867603 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:35.868580 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:35.868556 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:51.908161 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:51.908121 2583 patch_prober.go:28] interesting pod/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 12:40:51.908558 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:51.908189 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 12:40:53.960305 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.960267 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerID="cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987" exitCode=0 Jul 16 12:40:53.961015 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.960338 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerDied","Data":"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987"} Jul 16 12:40:53.961015 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.960739 2583 scope.go:117] "RemoveContainer" containerID="cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987" Jul 16 12:40:53.961746 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.961720 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:53.961818 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.961801 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:53.961859 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:53.961849 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 12:40:54.967807 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:54.967772 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerStarted","Data":"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7"} Jul 16 12:40:54.978971 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:54.978942 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:55.979025 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:55.978974 2583 patch_prober.go:28] interesting pod/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 12:40:55.979412 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:55.979054 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 12:40:56.978726 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:56.978692 2583 patch_prober.go:28] interesting pod/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 12:40:56.978911 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:56.978746 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 12:40:57.979149 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:57.979113 2583 patch_prober.go:28] interesting pod/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 12:40:57.979539 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:57.979167 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 12:40:58.979148 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:58.979111 2583 patch_prober.go:28] interesting pod/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 12:40:58.979325 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:58.979167 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 12:40:59.982392 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:59.982359 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:59.982782 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:59.982735 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:40:59.983457 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:40:59.983437 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:41:42.901101 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:41:42.901075 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:41:42.907451 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:41:42.907431 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:42:39.610121 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:39.610041 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:42:39.610569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:39.610276 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" containerID="cri-o://f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7" gracePeriod=30 Jul 16 12:42:43.090244 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.090199 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7"] Jul 16 12:42:43.093431 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.093411 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.095968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.095948 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-70ba56-serving-cert\"" Jul 16 12:42:43.105753 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.105726 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7"] Jul 16 12:42:43.188238 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.188198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.188461 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.188263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecc7197-96ad-4249-ab6b-c78e20af240f-openshift-service-ca\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.188461 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.188409 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzngm\" (UniqueName: \"kubernetes.io/projected/cecc7197-96ad-4249-ab6b-c78e20af240f-kube-api-access-dzngm\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.289175 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.289139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzngm\" (UniqueName: \"kubernetes.io/projected/cecc7197-96ad-4249-ab6b-c78e20af240f-kube-api-access-dzngm\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.289374 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.289198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.289374 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.289249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecc7197-96ad-4249-ab6b-c78e20af240f-openshift-service-ca\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.289374 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:42:43.289331 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-serving-cert: secret "maas-api-e2e-api-a-70ba56-serving-cert" not found Jul 16 12:42:43.289571 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:42:43.289399 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls podName:cecc7197-96ad-4249-ab6b-c78e20af240f nodeName:}" failed. No retries permitted until 2026-07-16 12:42:43.789379833 +0000 UTC m=+2161.483486537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls") pod "maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" (UID: "cecc7197-96ad-4249-ab6b-c78e20af240f") : secret "maas-api-e2e-api-a-70ba56-serving-cert" not found Jul 16 12:42:43.289956 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.289931 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecc7197-96ad-4249-ab6b-c78e20af240f-openshift-service-ca\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.301764 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.301741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzngm\" (UniqueName: \"kubernetes.io/projected/cecc7197-96ad-4249-ab6b-c78e20af240f-kube-api-access-dzngm\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.794465 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.794429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:43.796999 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:43.796968 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cecc7197-96ad-4249-ab6b-c78e20af240f-maas-api-tls\") pod \"maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7\" (UID: \"cecc7197-96ad-4249-ab6b-c78e20af240f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:44.005176 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:44.005141 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:44.142825 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:44.142800 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7"] Jul 16 12:42:44.145043 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:42:44.144958 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecc7197_96ad_4249_ab6b_c78e20af240f.slice/crio-14479314dd5bd0ac0392c5d78967ed0baa5dde7c8efcd1c85aa4ff8c4c384639 WatchSource:0}: Error finding container 14479314dd5bd0ac0392c5d78967ed0baa5dde7c8efcd1c85aa4ff8c4c384639: Status 404 returned error can't find the container with id 14479314dd5bd0ac0392c5d78967ed0baa5dde7c8efcd1c85aa4ff8c4c384639 Jul 16 12:42:44.146357 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:44.146337 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:42:44.501976 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:44.501933 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" event={"ID":"cecc7197-96ad-4249-ab6b-c78e20af240f","Type":"ContainerStarted","Data":"14479314dd5bd0ac0392c5d78967ed0baa5dde7c8efcd1c85aa4ff8c4c384639"} Jul 16 12:42:44.866478 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:44.866449 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:42:45.005115 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005092 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005252 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005133 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005252 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005182 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005252 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005213 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005252 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005241 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005284 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cttlw\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005356 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005391 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005450 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005426 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data\") pod \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\" (UID: \"bc2c3b59-429d-46e1-85bb-77d6b9d837b7\") " Jul 16 12:42:45.005655 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005457 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:42:45.005722 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005663 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:45.005800 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005767 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-credential-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.005856 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005805 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istiod-ca-cert\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.005915 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005887 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:45.006006 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.005621 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:45.006595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.006571 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:45.007418 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.007398 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:45.007485 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.007450 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 12:42:45.007926 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.007903 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw" (OuterVolumeSpecName: "kube-api-access-cttlw") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "kube-api-access-cttlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:42:45.007973 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.007935 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc2c3b59-429d-46e1-85bb-77d6b9d837b7" (UID: "bc2c3b59-429d-46e1-85bb-77d6b9d837b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107280 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107312 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-workload-socket\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107322 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttlw\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-kube-api-access-cttlw\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107333 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-podinfo\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107343 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-token\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107350 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-data\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.107375 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.107359 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2c3b59-429d-46e1-85bb-77d6b9d837b7-istio-envoy\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:42:45.509733 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.509696 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerID="f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7" exitCode=0 Jul 16 12:42:45.509733 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.509729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerDied","Data":"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7"} Jul 16 12:42:45.510304 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.509745 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" Jul 16 12:42:45.510304 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.509783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9" event={"ID":"bc2c3b59-429d-46e1-85bb-77d6b9d837b7","Type":"ContainerDied","Data":"da737630b7b1cbd0f7a1d7864ae086fb08120b13c60f2a2a209fdd4e90947b42"} Jul 16 12:42:45.510304 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.509807 2583 scope.go:117] "RemoveContainer" containerID="f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7" Jul 16 12:42:45.511427 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.511399 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" event={"ID":"cecc7197-96ad-4249-ab6b-c78e20af240f","Type":"ContainerStarted","Data":"ddec74054a5e53277582e60867bb83140a750180159f5a69d556de7620c37922"} Jul 16 12:42:45.511618 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.511595 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:42:45.519546 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.519531 2583 scope.go:117] "RemoveContainer" containerID="cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987" Jul 16 12:42:45.526944 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.526928 2583 scope.go:117] "RemoveContainer" containerID="f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7" Jul 16 12:42:45.527213 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:42:45.527196 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7\": container with ID starting with f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7 not found: ID does not exist" containerID="f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7" Jul 16 12:42:45.527271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.527222 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7"} err="failed to get container status \"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7\": rpc error: code = NotFound desc = could not find container \"f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7\": container with ID starting with f717c59f72972d9c5bb9d5e666c28ccf0fcfb08d05d9c124cdc5b9d2a5bf88a7 not found: ID does not exist" Jul 16 12:42:45.527271 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.527240 2583 scope.go:117] "RemoveContainer" containerID="cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987" Jul 16 12:42:45.527446 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:42:45.527432 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987\": container with ID starting with cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987 not found: ID does not exist" containerID="cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987" Jul 16 12:42:45.527490 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.527452 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987"} err="failed to get container status \"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987\": rpc error: code = NotFound desc = could not find container \"cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987\": container with ID starting with cd90fc74e5dd6543da24e4e09fe6e31c75bf039b4ea9e14890f412e14b2a1987 not found: ID does not exist" Jul 16 12:42:45.534148 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.534109 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" podStartSLOduration=1.713544588 podStartE2EDuration="2.534095162s" totalImagesPullingTime="820.550574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:42:45.533020264 +0000 UTC m=+2163.227127009" watchObservedRunningTime="2026-07-16 12:42:45.534095162 +0000 UTC m=+2163.228201881" Jul 16 12:42:45.546961 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.546938 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:42:45.553403 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:45.553379 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9ac0a7-openshift-default-647879757c-ctdj9"] Jul 16 12:42:46.795264 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:46.795230 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" path="/var/lib/kubelet/pods/bc2c3b59-429d-46e1-85bb-77d6b9d837b7/volumes" Jul 16 12:42:51.523234 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:42:51.523205 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7" Jul 16 12:43:35.478043 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.478008 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck"] Jul 16 12:43:35.478581 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.478541 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:35.478581 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.478561 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:35.478714 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.478646 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:35.480929 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.480910 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.483465 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.483446 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-cf65e4-serving-cert\"" Jul 16 12:43:35.492216 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.492185 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck"] Jul 16 12:43:35.655557 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.655525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbxj\" (UniqueName: \"kubernetes.io/projected/cba73f8b-0151-46ee-866f-bf50bb7eb520-kube-api-access-mhbxj\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.655731 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.655570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba73f8b-0151-46ee-866f-bf50bb7eb520-maas-api-tls\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.655731 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.655630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba73f8b-0151-46ee-866f-bf50bb7eb520-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.757209 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.757114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba73f8b-0151-46ee-866f-bf50bb7eb520-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.757383 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.757252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbxj\" (UniqueName: \"kubernetes.io/projected/cba73f8b-0151-46ee-866f-bf50bb7eb520-kube-api-access-mhbxj\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.757383 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.757307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba73f8b-0151-46ee-866f-bf50bb7eb520-maas-api-tls\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.760475 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.760445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cba73f8b-0151-46ee-866f-bf50bb7eb520-maas-api-tls\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.761127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.761103 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba73f8b-0151-46ee-866f-bf50bb7eb520-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:35.792396 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:35.792351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbxj\" (UniqueName: \"kubernetes.io/projected/cba73f8b-0151-46ee-866f-bf50bb7eb520-kube-api-access-mhbxj\") pod \"maas-api-e2e-shared-a-cf65e4-64879fd94-tctck\" (UID: \"cba73f8b-0151-46ee-866f-bf50bb7eb520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:36.091577 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.091492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:36.229061 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.229036 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck"] Jul 16 12:43:36.231540 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:43:36.231517 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba73f8b_0151_46ee_866f_bf50bb7eb520.slice/crio-91cbd95a31dc63ca77b5f2ff8f426fcd26d1bcda397226895fe3ddffb44ce48a WatchSource:0}: Error finding container 91cbd95a31dc63ca77b5f2ff8f426fcd26d1bcda397226895fe3ddffb44ce48a: Status 404 returned error can't find the container with id 91cbd95a31dc63ca77b5f2ff8f426fcd26d1bcda397226895fe3ddffb44ce48a Jul 16 12:43:36.771335 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.771301 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" event={"ID":"cba73f8b-0151-46ee-866f-bf50bb7eb520","Type":"ContainerStarted","Data":"f0e50f1350c0c2d5a37f7a590838f2f34d8052a62ad5624d4df7fcb8e2f2c9da"} Jul 16 12:43:36.771724 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.771349 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" event={"ID":"cba73f8b-0151-46ee-866f-bf50bb7eb520","Type":"ContainerStarted","Data":"91cbd95a31dc63ca77b5f2ff8f426fcd26d1bcda397226895fe3ddffb44ce48a"} Jul 16 12:43:36.771724 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.771398 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:36.793431 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:36.793338 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" podStartSLOduration=1.4843007830000001 podStartE2EDuration="1.793322576s" totalImagesPullingTime="309.021793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:43:36.791215916 +0000 UTC m=+2214.485322659" watchObservedRunningTime="2026-07-16 12:43:36.793322576 +0000 UTC m=+2214.487429296" Jul 16 12:43:42.781402 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:42.781373 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-cf65e4-64879fd94-tctck" Jul 16 12:43:43.494550 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.494510 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4"] Jul 16 12:43:43.495110 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.495093 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:43.495172 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.495113 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:43.495235 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.495225 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2c3b59-429d-46e1-85bb-77d6b9d837b7" containerName="istio-proxy" Jul 16 12:43:43.498752 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.498734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.502351 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.502333 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-3c72a3-serving-cert\"" Jul 16 12:43:43.511030 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.511009 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4"] Jul 16 12:43:43.631761 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.631728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8ft\" (UniqueName: \"kubernetes.io/projected/fbfe15b7-094d-454a-8d43-4e4853c51933-kube-api-access-zt8ft\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.631968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.631772 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe15b7-094d-454a-8d43-4e4853c51933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.631968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.631889 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.732772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.732682 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.732772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.732740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8ft\" (UniqueName: \"kubernetes.io/projected/fbfe15b7-094d-454a-8d43-4e4853c51933-kube-api-access-zt8ft\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.732772 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.732766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe15b7-094d-454a-8d43-4e4853c51933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.733083 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:43:43.732825 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-serving-cert: secret "maas-api-e2e-shared-b-3c72a3-serving-cert" not found Jul 16 12:43:43.733083 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:43:43.732904 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls podName:fbfe15b7-094d-454a-8d43-4e4853c51933 nodeName:}" failed. No retries permitted until 2026-07-16 12:43:44.232887487 +0000 UTC m=+2221.926994211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls") pod "maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" (UID: "fbfe15b7-094d-454a-8d43-4e4853c51933") : secret "maas-api-e2e-shared-b-3c72a3-serving-cert" not found Jul 16 12:43:43.733398 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.733379 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe15b7-094d-454a-8d43-4e4853c51933-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:43.744103 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:43.744070 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8ft\" (UniqueName: \"kubernetes.io/projected/fbfe15b7-094d-454a-8d43-4e4853c51933-kube-api-access-zt8ft\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:44.236636 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:44.236603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:44.239265 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:44.239232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbfe15b7-094d-454a-8d43-4e4853c51933-maas-api-tls\") pod \"maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4\" (UID: \"fbfe15b7-094d-454a-8d43-4e4853c51933\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:44.409426 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:44.409397 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:44.551543 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:44.551394 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4"] Jul 16 12:43:44.554041 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:43:44.554014 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfe15b7_094d_454a_8d43_4e4853c51933.slice/crio-5432abffe69c6bb27d1c2a2051e50e447173b6ca30844f54bba3cd87829b82da WatchSource:0}: Error finding container 5432abffe69c6bb27d1c2a2051e50e447173b6ca30844f54bba3cd87829b82da: Status 404 returned error can't find the container with id 5432abffe69c6bb27d1c2a2051e50e447173b6ca30844f54bba3cd87829b82da Jul 16 12:43:44.809051 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:44.808953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" event={"ID":"fbfe15b7-094d-454a-8d43-4e4853c51933","Type":"ContainerStarted","Data":"5432abffe69c6bb27d1c2a2051e50e447173b6ca30844f54bba3cd87829b82da"} Jul 16 12:43:45.814946 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:45.814910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" event={"ID":"fbfe15b7-094d-454a-8d43-4e4853c51933","Type":"ContainerStarted","Data":"cfd552c9c3e23740a129b987a9eda7216d916d79282f35b57159fabc392120e8"} Jul 16 12:43:45.815356 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:45.815057 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:43:45.841624 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:45.841570 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" podStartSLOduration=2.177411763 podStartE2EDuration="2.841554779s" totalImagesPullingTime="664.143016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:43:45.839126841 +0000 UTC m=+2223.533233590" watchObservedRunningTime="2026-07-16 12:43:45.841554779 +0000 UTC m=+2223.535661498" Jul 16 12:43:51.824661 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:43:51.824632 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4" Jul 16 12:44:08.704040 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.703950 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:44:08.706970 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.706951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.710784 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.710756 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"auth-test-model-cf65e4-kserve-self-signed-certs\"" Jul 16 12:44:08.710910 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.710818 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"openshift-service-ca.crt\"" Jul 16 12:44:08.710910 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.710834 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"default-dockercfg-4z2m4\"" Jul 16 12:44:08.710910 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.710894 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"kube-root-ca.crt\"" Jul 16 12:44:08.719206 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.719181 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:44:08.858680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvhf\" (UniqueName: \"kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.858853 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.858853 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.858853 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.858853 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.859033 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.858863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959410 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvhf\" (UniqueName: \"kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959491 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959519 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959745 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.959984 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959958 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.960083 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.959964 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.960083 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.960027 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.961897 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.961876 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.962003 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.961973 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:08.969683 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:08.969664 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvhf\" (UniqueName: \"kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf\") pod \"auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:09.016269 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:09.016244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:09.152763 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:09.152737 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:44:09.154799 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:44:09.154770 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4b3d2b_f5cc_4945_90fa_99a07a863bfb.slice/crio-6bd1ef926923409640c78e35299f9665362af7f237edb5bc0793dee576c9e51b WatchSource:0}: Error finding container 6bd1ef926923409640c78e35299f9665362af7f237edb5bc0793dee576c9e51b: Status 404 returned error can't find the container with id 6bd1ef926923409640c78e35299f9665362af7f237edb5bc0793dee576c9e51b Jul 16 12:44:09.928258 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:09.928219 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" event={"ID":"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb","Type":"ContainerStarted","Data":"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2"} Jul 16 12:44:09.928657 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:09.928270 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" event={"ID":"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb","Type":"ContainerStarted","Data":"6bd1ef926923409640c78e35299f9665362af7f237edb5bc0793dee576c9e51b"} Jul 16 12:44:09.948855 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:09.948803 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" podStartSLOduration=1.948788935 podStartE2EDuration="1.948788935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:09.948501673 +0000 UTC m=+2247.642608391" watchObservedRunningTime="2026-07-16 12:44:09.948788935 +0000 UTC m=+2247.642895655" Jul 16 12:44:19.016562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:19.016526 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:19.016562 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:19.016568 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:19.031187 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:19.031165 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:44:19.987645 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:44:19.987612 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:46:42.927857 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:46:42.927830 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:46:42.934947 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:46:42.934924 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rd4b2_e4aa1f4d-445c-4753-a5dc-c7c5317fd337/ovn-acl-logging/0.log" Jul 16 12:46:50.693572 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:46:50.693532 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:47:05.713448 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:05.713364 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" podUID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" containerName="main" containerID="cri-o://96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2" gracePeriod=45 Jul 16 12:47:05.977710 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:05.977649 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:47:06.018822 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.018792 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.018967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.018840 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zvhf\" (UniqueName: \"kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.018967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.018895 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.018967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.018924 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.018967 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.018965 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.019191 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.019001 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir\") pod \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\" (UID: \"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb\") " Jul 16 12:47:06.019297 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.019235 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home" (OuterVolumeSpecName: "home") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:06.019297 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.019265 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:06.019378 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.019298 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:06.020922 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.020892 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm" (OuterVolumeSpecName: "dshm") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:47:06.021065 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.021040 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf" (OuterVolumeSpecName: "kube-api-access-9zvhf") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "kube-api-access-9zvhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:06.021129 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.021047 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" (UID: "fb4b3d2b-f5cc-4945-90fa-99a07a863bfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:06.119643 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119616 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tls-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.119643 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119642 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvhf\" (UniqueName: \"kubernetes.io/projected/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-kube-api-access-9zvhf\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.119785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119650 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-dshm\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.119785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119659 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-model-cache\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.119785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119666 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-home\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.119785 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.119692 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb-tmp-dir\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:47:06.400827 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.400801 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:47:06.401228 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.401215 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" containerName="main" Jul 16 12:47:06.401282 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.401229 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" containerName="main" Jul 16 12:47:06.401318 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.401303 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" containerName="main" Jul 16 12:47:06.404356 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.404342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.406784 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.406765 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cf65e4\"/\"sub-test-model-cf65e4-kserve-self-signed-certs\"" Jul 16 12:47:06.414854 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.414830 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:47:06.421237 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421217 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.421324 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.421324 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.421324 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctncs\" (UniqueName: \"kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.421451 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.421451 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.421394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.521759 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.521729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.521899 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.521766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.521899 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.521781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.521894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctncs\" (UniqueName: \"kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522019 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.521982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.522032 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522181 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.522151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522239 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.522228 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.522371 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.522352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.524022 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.524005 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.524274 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.524257 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.530276 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.530259 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctncs\" (UniqueName: \"kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs\") pod \"sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.713807 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.713734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:06.780595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.780563 2583 generic.go:356] "Generic (PLEG): container finished" podID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" containerID="96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2" exitCode=0 Jul 16 12:47:06.780737 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.780646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" event={"ID":"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb","Type":"ContainerDied","Data":"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2"} Jul 16 12:47:06.780737 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.780673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" event={"ID":"fb4b3d2b-f5cc-4945-90fa-99a07a863bfb","Type":"ContainerDied","Data":"6bd1ef926923409640c78e35299f9665362af7f237edb5bc0793dee576c9e51b"} Jul 16 12:47:06.780737 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.780689 2583 scope.go:117] "RemoveContainer" containerID="96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2" Jul 16 12:47:06.780834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.780805 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv" Jul 16 12:47:06.791172 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.791148 2583 scope.go:117] "RemoveContainer" containerID="96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2" Jul 16 12:47:06.791429 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:47:06.791403 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2\": container with ID starting with 96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2 not found: ID does not exist" containerID="96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2" Jul 16 12:47:06.791516 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.791439 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2"} err="failed to get container status \"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2\": rpc error: code = NotFound desc = could not find container \"96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2\": container with ID starting with 96f1e5cc771460c2dc75888833b84134879cb9e8563624459dc63b322682b9e2 not found: ID does not exist" Jul 16 12:47:06.805448 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.805421 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:47:06.810204 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.810182 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/auth-test-model-cf65e4-kserve-6864f99cd7-gsnxv"] Jul 16 12:47:06.851731 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:06.851698 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:47:06.854537 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:47:06.854514 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacb7c29b_4b57_4f4a_ad29_07ffa2373bec.slice/crio-4fe840f937bd7f5e85ebdcfc2d30ca56ba109bf5c1b8bdfeb4c8892d126e87d4 WatchSource:0}: Error finding container 4fe840f937bd7f5e85ebdcfc2d30ca56ba109bf5c1b8bdfeb4c8892d126e87d4: Status 404 returned error can't find the container with id 4fe840f937bd7f5e85ebdcfc2d30ca56ba109bf5c1b8bdfeb4c8892d126e87d4 Jul 16 12:47:07.787489 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:07.787459 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" event={"ID":"acb7c29b-4b57-4f4a-ad29-07ffa2373bec","Type":"ContainerStarted","Data":"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35"} Jul 16 12:47:07.787886 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:07.787499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" event={"ID":"acb7c29b-4b57-4f4a-ad29-07ffa2373bec","Type":"ContainerStarted","Data":"4fe840f937bd7f5e85ebdcfc2d30ca56ba109bf5c1b8bdfeb4c8892d126e87d4"} Jul 16 12:47:07.810090 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:07.810047 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" podStartSLOduration=1.810033667 podStartE2EDuration="1.810033667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:47:07.807891593 +0000 UTC m=+2425.501998357" watchObservedRunningTime="2026-07-16 12:47:07.810033667 +0000 UTC m=+2425.504140384" Jul 16 12:47:08.794965 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:08.794930 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4b3d2b-f5cc-4945-90fa-99a07a863bfb" path="/var/lib/kubelet/pods/fb4b3d2b-f5cc-4945-90fa-99a07a863bfb/volumes" Jul 16 12:47:16.714905 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:16.714865 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:16.715522 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:16.714917 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:16.727125 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:16.727101 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:47:16.839635 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:47:16.839612 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:48:07.218539 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:07.218509 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:48:22.237912 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.237854 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" podUID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" containerName="main" containerID="cri-o://b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35" gracePeriod=45 Jul 16 12:48:22.486802 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.486777 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:48:22.530652 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530589 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.530789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530649 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctncs\" (UniqueName: \"kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.530789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530689 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.530789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530748 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.530789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530779 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.531013 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530832 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache" (OuterVolumeSpecName: "model-cache") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:22.531013 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.530844 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm\") pod \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\" (UID: \"acb7c29b-4b57-4f4a-ad29-07ffa2373bec\") " Jul 16 12:48:22.531127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.531034 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home" (OuterVolumeSpecName: "home") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:22.531241 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.531221 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:22.531327 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.531304 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-model-cache\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:22.531394 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.531333 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-home\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:22.532799 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.532771 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs" (OuterVolumeSpecName: "kube-api-access-ctncs") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "kube-api-access-ctncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:48:22.533047 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.533031 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm" (OuterVolumeSpecName: "dshm") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:22.533237 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.533211 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acb7c29b-4b57-4f4a-ad29-07ffa2373bec" (UID: "acb7c29b-4b57-4f4a-ad29-07ffa2373bec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:22.632182 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.632157 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tls-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:22.632182 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.632180 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-tmp-dir\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:22.632344 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.632188 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-dshm\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:22.632344 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:22.632196 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctncs\" (UniqueName: \"kubernetes.io/projected/acb7c29b-4b57-4f4a-ad29-07ffa2373bec-kube-api-access-ctncs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:48:23.147632 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.147604 2583 generic.go:356] "Generic (PLEG): container finished" podID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" containerID="b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35" exitCode=0 Jul 16 12:48:23.147811 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.147648 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" Jul 16 12:48:23.147811 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.147691 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" event={"ID":"acb7c29b-4b57-4f4a-ad29-07ffa2373bec","Type":"ContainerDied","Data":"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35"} Jul 16 12:48:23.147811 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.147740 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2" event={"ID":"acb7c29b-4b57-4f4a-ad29-07ffa2373bec","Type":"ContainerDied","Data":"4fe840f937bd7f5e85ebdcfc2d30ca56ba109bf5c1b8bdfeb4c8892d126e87d4"} Jul 16 12:48:23.147811 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.147761 2583 scope.go:117] "RemoveContainer" containerID="b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35" Jul 16 12:48:23.156710 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.156689 2583 scope.go:117] "RemoveContainer" containerID="b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35" Jul 16 12:48:23.157128 ip-10-0-139-22 kubenswrapper[2583]: E0716 12:48:23.156975 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35\": container with ID starting with b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35 not found: ID does not exist" containerID="b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35" Jul 16 12:48:23.157225 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.157141 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35"} err="failed to get container status \"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35\": rpc error: code = NotFound desc = could not find container \"b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35\": container with ID starting with b4227c23061f41476877263e118da02c6c6654995d302720c79999909ed36e35 not found: ID does not exist" Jul 16 12:48:23.170721 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.170700 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:48:23.176384 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:23.176364 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cf65e4/sub-test-model-cf65e4-kserve-9fb49d5cc-wk4k2"] Jul 16 12:48:24.794202 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:24.794171 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" path="/var/lib/kubelet/pods/acb7c29b-4b57-4f4a-ad29-07ffa2373bec/volumes" Jul 16 12:48:31.802276 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.802244 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:48:31.802834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.802806 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" containerName="main" Jul 16 12:48:31.802834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.802827 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" containerName="main" Jul 16 12:48:31.802960 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.802942 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb7c29b-4b57-4f4a-ad29-07ffa2373bec" containerName="main" Jul 16 12:48:31.805902 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.805881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.809834 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.809816 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"kube-root-ca.crt\"" Jul 16 12:48:31.809942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.809829 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"openshift-service-ca.crt\"" Jul 16 12:48:31.809942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.809839 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"default-dockercfg-j2g8z\"" Jul 16 12:48:31.809942 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.809851 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3c72a3\"/\"rate-test-model-3c72a3-kserve-self-signed-certs\"" Jul 16 12:48:31.816052 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.816033 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:48:31.907381 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.907509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.907509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907406 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.907509 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.907622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907539 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:31.907622 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:31.907566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trm9w\" (UniqueName: \"kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008178 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008178 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008405 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008405 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008515 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trm9w\" (UniqueName: \"kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008623 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008565 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008666 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008616 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.008828 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.008810 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.010372 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.010342 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.010587 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.010572 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.018644 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.018621 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trm9w\" (UniqueName: \"kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w\") pod \"rate-test-model-3c72a3-kserve-844767d86c-fb2pn\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.115235 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.115165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:32.246469 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:32.246433 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:48:32.249364 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:48:32.249334 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod387fbf1a_863f_4031_bd7d_9789628b3791.slice/crio-1f4518a68886a6e7ffb2fb989c99002fec4a3a7212b6f32d727e375c3c34d7dd WatchSource:0}: Error finding container 1f4518a68886a6e7ffb2fb989c99002fec4a3a7212b6f32d727e375c3c34d7dd: Status 404 returned error can't find the container with id 1f4518a68886a6e7ffb2fb989c99002fec4a3a7212b6f32d727e375c3c34d7dd Jul 16 12:48:33.198428 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:33.198391 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" event={"ID":"387fbf1a-863f-4031-bd7d-9789628b3791","Type":"ContainerStarted","Data":"1363130e16cc6cb156ff5fb2a4f9f54b1703b017fbd6b2daec25ec96ee5d088c"} Jul 16 12:48:33.198428 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:33.198431 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" event={"ID":"387fbf1a-863f-4031-bd7d-9789628b3791","Type":"ContainerStarted","Data":"1f4518a68886a6e7ffb2fb989c99002fec4a3a7212b6f32d727e375c3c34d7dd"} Jul 16 12:48:33.219408 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:33.219363 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" podStartSLOduration=2.219350308 podStartE2EDuration="2.219350308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:48:33.217585507 +0000 UTC m=+2510.911692226" watchObservedRunningTime="2026-07-16 12:48:33.219350308 +0000 UTC m=+2510.913457070" Jul 16 12:48:42.116068 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:42.115975 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:42.116068 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:42.116032 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:42.129455 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:42.129433 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:48:42.252320 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:48:42.252300 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:49:35.398583 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:35.398551 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:49:50.417941 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.417881 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" podUID="387fbf1a-863f-4031-bd7d-9789628b3791" containerName="main" containerID="cri-o://1363130e16cc6cb156ff5fb2a4f9f54b1703b017fbd6b2daec25ec96ee5d088c" gracePeriod=45 Jul 16 12:49:50.559646 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.559610 2583 generic.go:356] "Generic (PLEG): container finished" podID="387fbf1a-863f-4031-bd7d-9789628b3791" containerID="1363130e16cc6cb156ff5fb2a4f9f54b1703b017fbd6b2daec25ec96ee5d088c" exitCode=0 Jul 16 12:49:50.559771 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.559682 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" event={"ID":"387fbf1a-863f-4031-bd7d-9789628b3791","Type":"ContainerDied","Data":"1363130e16cc6cb156ff5fb2a4f9f54b1703b017fbd6b2daec25ec96ee5d088c"} Jul 16 12:49:50.659131 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.659112 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:49:50.764453 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764385 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764453 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764414 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764453 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764437 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764706 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764479 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trm9w\" (UniqueName: \"kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764706 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764505 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764706 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764544 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache\") pod \"387fbf1a-863f-4031-bd7d-9789628b3791\" (UID: \"387fbf1a-863f-4031-bd7d-9789628b3791\") " Jul 16 12:49:50.764706 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764691 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home" (OuterVolumeSpecName: "home") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:50.764899 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764776 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:50.764899 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.764873 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache" (OuterVolumeSpecName: "model-cache") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:50.766584 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.766557 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:49:50.766686 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.766606 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm" (OuterVolumeSpecName: "dshm") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:49:50.766686 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.766649 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w" (OuterVolumeSpecName: "kube-api-access-trm9w") pod "387fbf1a-863f-4031-bd7d-9789628b3791" (UID: "387fbf1a-863f-4031-bd7d-9789628b3791"). InnerVolumeSpecName "kube-api-access-trm9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:49:50.865411 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865385 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-tmp-dir\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.865411 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865406 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-model-cache\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.865411 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865415 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387fbf1a-863f-4031-bd7d-9789628b3791-tls-certs\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.865606 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865425 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-home\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.865606 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865437 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387fbf1a-863f-4031-bd7d-9789628b3791-dshm\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:50.865606 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:50.865446 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trm9w\" (UniqueName: \"kubernetes.io/projected/387fbf1a-863f-4031-bd7d-9789628b3791-kube-api-access-trm9w\") on node \"ip-10-0-139-22.ec2.internal\" DevicePath \"\"" Jul 16 12:49:51.566582 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:51.566545 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" Jul 16 12:49:51.566582 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:51.566563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn" event={"ID":"387fbf1a-863f-4031-bd7d-9789628b3791","Type":"ContainerDied","Data":"1f4518a68886a6e7ffb2fb989c99002fec4a3a7212b6f32d727e375c3c34d7dd"} Jul 16 12:49:51.567106 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:51.566607 2583 scope.go:117] "RemoveContainer" containerID="1363130e16cc6cb156ff5fb2a4f9f54b1703b017fbd6b2daec25ec96ee5d088c" Jul 16 12:49:51.585878 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:51.585850 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:49:51.593270 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:51.593250 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3c72a3/rate-test-model-3c72a3-kserve-844767d86c-fb2pn"] Jul 16 12:49:52.795010 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:49:52.794969 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="387fbf1a-863f-4031-bd7d-9789628b3791" path="/var/lib/kubelet/pods/387fbf1a-863f-4031-bd7d-9789628b3791/volumes" Jul 16 12:50:16.122566 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.122496 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5czs6_3ca17d96-66a9-4bcf-8783-e7cf20ca6545/manager/0.log" Jul 16 12:50:16.250308 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.250285 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bw7dw_7fba93b3-1602-4b38-901b-ee0facca0567/manager/0.log" Jul 16 12:50:16.388013 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.387943 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dcdc7c8b8-7c9zr_7c86bdd7-9ba4-46fa-8684-831706f7a3f2/manager/0.log" Jul 16 12:50:16.524650 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.524628 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-52ws8_684cf32e-3332-4065-973e-70aa82907ae4/server/0.log" Jul 16 12:50:16.657484 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.657437 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4cqq4_51a5f9f9-eb1f-48a9-b914-622742efb141/manager/0.log" Jul 16 12:50:16.790630 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.790612 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx_e641c5e4-bc91-4b0e-96b8-3b0f9265150f/copy-manifests/0.log" Jul 16 12:50:16.803726 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:16.803704 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx_e641c5e4-bc91-4b0e-96b8-3b0f9265150f/manager/0.log" Jul 16 12:50:17.512323 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:17.512301 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d96948d97-pbz79_a863f304-5004-47fd-8ac7-7ca10f4221b7/maas-api/0.log" Jul 16 12:50:17.777049 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:17.776945 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-api-a-70ba56-5fc8868447-fjgc7_cecc7197-96ad-4249-ab6b-c78e20af240f/maas-api/0.log" Jul 16 12:50:18.040555 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:18.040488 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-derive-15231c4d-67677895c4-p55xj_41a84a17-805d-4cfe-be35-b32a43ecfcf8/maas-api/0.log" Jul 16 12:50:18.675730 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:18.675694 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-cf65e4-64879fd94-tctck_cba73f8b-0151-46ee-866f-bf50bb7eb520/maas-api/0.log" Jul 16 12:50:18.799231 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:18.799199 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-3c72a3-bb894897f-s8kw4_fbfe15b7-094d-454a-8d43-4e4853c51933/maas-api/0.log" Jul 16 12:50:20.401680 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:20.401655 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-99tv7_ec31fbdb-59c7-4502-a187-7a3976e75ed7/postgres/0.log" Jul 16 12:50:21.999127 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:21.999099 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9rs8g_1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc/manager/0.log" Jul 16 12:50:22.130381 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:22.130353 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j4pq2_9e850581-6bce-4f18-90e8-e1f2e4026069/manager/0.log" Jul 16 12:50:22.259914 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:22.259846 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nsqtk_0f7db5f6-202b-4e57-92a3-35a1a6472546/kuadrant-console-plugin/0.log" Jul 16 12:50:22.395947 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:22.395917 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z5rbn_8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c/registry-server/0.log" Jul 16 12:50:22.797279 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:22.797258 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-d9mhc_bee8e9ae-f107-4d83-a256-62f367dd518a/manager/0.log" Jul 16 12:50:23.225600 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:23.225579 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49pz5j_d701f5ac-2bbf-4073-9668-e22bface55b5/istio-proxy/0.log" Jul 16 12:50:23.600792 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:23.600729 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-wzwpq_4742ae0a-cc41-445f-b477-526b52f9093e/kube-auth-proxy/0.log" Jul 16 12:50:23.731741 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:23.731714 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/1.log" Jul 16 12:50:23.983305 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:23.983277 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hcw56_525e4580-d118-4b19-9b09-9417fc42c178/payload-processing/0.log" Jul 16 12:50:24.609241 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:24.609215 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7z4sj_47b002ee-14bd-4ec5-919f-0b8c3e8fa677/main/0.log" Jul 16 12:50:24.735465 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:24.735428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qnxlc_754ad6ca-f51a-4603-aae0-56f243b24ade/main/0.log" Jul 16 12:50:24.988415 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:24.988393 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-v9k2b_e45db0f5-c580-4237-b82d-4670b51665f9/main/0.log" Jul 16 12:50:34.544926 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:34.544897 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bzpqj_470371ed-667c-4545-a836-883ce8e71ff5/global-pull-secret-syncer/0.log" Jul 16 12:50:34.737098 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:34.737066 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dsmnv_75b86a55-a963-41ff-82a1-c528689346a3/konnectivity-agent/0.log" Jul 16 12:50:34.795310 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:34.795247 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-22.ec2.internal_7ae3cb7d0f0636b302c0e293270dd787/haproxy/0.log" Jul 16 12:50:39.835534 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:39.835506 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9rs8g_1c81d8a4-14fb-474a-bd82-6d7fa5baf0bc/manager/0.log" Jul 16 12:50:39.870817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:39.870791 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-j4pq2_9e850581-6bce-4f18-90e8-e1f2e4026069/manager/0.log" Jul 16 12:50:39.897817 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:39.897796 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nsqtk_0f7db5f6-202b-4e57-92a3-35a1a6472546/kuadrant-console-plugin/0.log" Jul 16 12:50:39.940956 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:39.940939 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z5rbn_8a2eaa1b-1d25-40eb-bb3f-fd0c9912759c/registry-server/0.log" Jul 16 12:50:40.229746 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:40.229717 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-d9mhc_bee8e9ae-f107-4d83-a256-62f367dd518a/manager/0.log" Jul 16 12:50:42.441694 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:42.441665 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnl6v_8b1abf52-0d38-4c7f-b0b6-199a57a787fd/node-exporter/0.log" Jul 16 12:50:42.468490 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:42.468471 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnl6v_8b1abf52-0d38-4c7f-b0b6-199a57a787fd/kube-rbac-proxy/0.log" Jul 16 12:50:42.493089 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:42.493069 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnl6v_8b1abf52-0d38-4c7f-b0b6-199a57a787fd/init-textfile/0.log" Jul 16 12:50:44.144315 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.144281 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9"] Jul 16 12:50:44.144854 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.144839 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="387fbf1a-863f-4031-bd7d-9789628b3791" containerName="main" Jul 16 12:50:44.144904 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.144857 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="387fbf1a-863f-4031-bd7d-9789628b3791" containerName="main" Jul 16 12:50:44.144968 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.144957 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="387fbf1a-863f-4031-bd7d-9789628b3791" containerName="main" Jul 16 12:50:44.150523 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.150502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.153357 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.153234 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xm7nj\"/\"default-dockercfg-7f7h2\"" Jul 16 12:50:44.154337 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.154295 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xm7nj\"/\"openshift-service-ca.crt\"" Jul 16 12:50:44.154485 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.154408 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xm7nj\"/\"kube-root-ca.crt\"" Jul 16 12:50:44.156061 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.156040 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9"] Jul 16 12:50:44.263118 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.263096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-lib-modules\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.263250 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.263127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7cj\" (UniqueName: \"kubernetes.io/projected/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-kube-api-access-zm7cj\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.263250 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.263161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-podres\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.263250 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.263204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-proc\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.263402 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.263251 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-sys\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364437 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-proc\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-sys\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364537 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-proc\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364569 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-lib-modules\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7cj\" (UniqueName: \"kubernetes.io/projected/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-kube-api-access-zm7cj\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364558 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-sys\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364615 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-podres\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364634 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-lib-modules\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.364732 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.364694 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-podres\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.374185 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.374163 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7cj\" (UniqueName: \"kubernetes.io/projected/735f71d9-6e23-4ad5-bbbd-59a221aa62cb-kube-api-access-zm7cj\") pod \"perf-node-gather-daemonset-q7zd9\" (UID: \"735f71d9-6e23-4ad5-bbbd-59a221aa62cb\") " pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.461630 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.461579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.589734 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.589681 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9"] Jul 16 12:50:44.591971 ip-10-0-139-22 kubenswrapper[2583]: W0716 12:50:44.591946 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod735f71d9_6e23_4ad5_bbbd_59a221aa62cb.slice/crio-c0ea24b93862d02cd49582050484937c1a4a6721a6f74eee045a30b0f55b662f WatchSource:0}: Error finding container c0ea24b93862d02cd49582050484937c1a4a6721a6f74eee045a30b0f55b662f: Status 404 returned error can't find the container with id c0ea24b93862d02cd49582050484937c1a4a6721a6f74eee045a30b0f55b662f Jul 16 12:50:44.817621 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.817587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" event={"ID":"735f71d9-6e23-4ad5-bbbd-59a221aa62cb","Type":"ContainerStarted","Data":"a58782b4fda5234b080bf50ec5f4dee7fdab134b22193bd0d101030875c8df8b"} Jul 16 12:50:44.817742 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.817647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" event={"ID":"735f71d9-6e23-4ad5-bbbd-59a221aa62cb","Type":"ContainerStarted","Data":"c0ea24b93862d02cd49582050484937c1a4a6721a6f74eee045a30b0f55b662f"} Jul 16 12:50:44.817742 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.817677 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:44.835789 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:44.835745 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" podStartSLOduration=0.835730441 podStartE2EDuration="835.730441ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:44.834806377 +0000 UTC m=+2642.528913095" watchObservedRunningTime="2026-07-16 12:50:44.835730441 +0000 UTC m=+2642.529837159" Jul 16 12:50:47.207098 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:47.207067 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggfxd_e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3/dns/0.log" Jul 16 12:50:47.264766 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:47.264741 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggfxd_e0e196ea-dfbd-41e9-9b2d-3b18e000c6e3/kube-rbac-proxy/0.log" Jul 16 12:50:47.454595 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:47.454570 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbpk4_1d320534-8242-4791-8292-7fbfca21d331/dns-node-resolver/0.log" Jul 16 12:50:48.162978 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:48.162946 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kb2qd_2b6e0e6e-b9ae-454b-8c0f-ca76ccf970a7/node-ca/0.log" Jul 16 12:50:49.176915 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:49.176880 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49pz5j_d701f5ac-2bbf-4073-9668-e22bface55b5/istio-proxy/0.log" Jul 16 12:50:49.356921 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:49.356897 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-wzwpq_4742ae0a-cc41-445f-b477-526b52f9093e/kube-auth-proxy/0.log" Jul 16 12:50:49.390192 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:49.390164 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/0.log" Jul 16 12:50:49.416794 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:49.416771 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/1.log" Jul 16 12:50:49.521860 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:49.521795 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hcw56_525e4580-d118-4b19-9b09-9417fc42c178/payload-processing/0.log" Jul 16 12:50:50.296277 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:50.296252 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qt9pz_de45197d-cd2f-4624-b972-cc76efc7ed86/serve-healthcheck-canary/0.log" Jul 16 12:50:50.832557 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:50.832531 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xm7nj/perf-node-gather-daemonset-q7zd9" Jul 16 12:50:50.834722 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:50.834699 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj2mx_4396a3f0-4278-4947-a03a-eca9e09dff08/kube-rbac-proxy/0.log" Jul 16 12:50:50.859928 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:50.859904 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj2mx_4396a3f0-4278-4947-a03a-eca9e09dff08/exporter/0.log" Jul 16 12:50:50.881216 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:50.881191 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jj2mx_4396a3f0-4278-4947-a03a-eca9e09dff08/extractor/0.log" Jul 16 12:50:53.460175 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.460144 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5czs6_3ca17d96-66a9-4bcf-8783-e7cf20ca6545/manager/0.log" Jul 16 12:50:53.570104 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.570077 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bw7dw_7fba93b3-1602-4b38-901b-ee0facca0567/manager/0.log" Jul 16 12:50:53.648952 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.648925 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dcdc7c8b8-7c9zr_7c86bdd7-9ba4-46fa-8684-831706f7a3f2/manager/0.log" Jul 16 12:50:53.674788 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.674766 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-52ws8_684cf32e-3332-4065-973e-70aa82907ae4/server/0.log" Jul 16 12:50:53.736133 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.736082 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4cqq4_51a5f9f9-eb1f-48a9-b914-622742efb141/manager/0.log" Jul 16 12:50:53.812972 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.812949 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx_e641c5e4-bc91-4b0e-96b8-3b0f9265150f/manager/0.log" Jul 16 12:50:53.854049 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:53.854032 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9bf88fc-4g9tx_e641c5e4-bc91-4b0e-96b8-3b0f9265150f/copy-manifests/0.log" Jul 16 12:50:54.404699 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:54.404636 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49pz5j_d701f5ac-2bbf-4073-9668-e22bface55b5/istio-proxy/0.log" Jul 16 12:50:54.551527 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:54.551505 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-wzwpq_4742ae0a-cc41-445f-b477-526b52f9093e/kube-auth-proxy/0.log" Jul 16 12:50:54.571830 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:54.571805 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/0.log" Jul 16 12:50:54.598034 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:54.598010 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/1.log" Jul 16 12:50:54.638599 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:54.638580 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hcw56_525e4580-d118-4b19-9b09-9417fc42c178/payload-processing/0.log" Jul 16 12:50:55.874667 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:55.874644 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49pz5j_d701f5ac-2bbf-4073-9668-e22bface55b5/istio-proxy/0.log" Jul 16 12:50:56.073858 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:56.073829 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569f4f6548-wzwpq_4742ae0a-cc41-445f-b477-526b52f9093e/kube-auth-proxy/0.log" Jul 16 12:50:56.104414 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:56.104375 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/0.log" Jul 16 12:50:56.130363 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:56.130303 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xv9d5_dfbdeff0-9200-4c94-b0a2-845ecff1fa96/istio-proxy/1.log" Jul 16 12:50:56.232425 ip-10-0-139-22 kubenswrapper[2583]: I0716 12:50:56.232405 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-hcw56_525e4580-d118-4b19-9b09-9417fc42c178/payload-processing/0.log"