Jul 10 13:10:02.351337 ip-10-0-128-249 systemd[1]: Starting Kubernetes Kubelet... Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.760787 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766209 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766223 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766228 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766233 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766238 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766242 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766246 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:02.773883 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766250 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766255 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766259 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766262 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766267 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766270 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766275 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766279 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766283 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766289 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766294 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766299 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766304 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766308 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766312 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766316 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766320 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766324 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766329 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766333 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:02.783745 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766337 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766341 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766353 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766357 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766361 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766365 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766371 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766375 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766380 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766384 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766389 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766393 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766397 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766402 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766406 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766411 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766415 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766419 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766423 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:02.784567 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766429 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766434 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766438 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766442 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766446 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766449 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766455 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766460 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766465 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766469 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766473 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766477 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766481 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766485 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766488 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766492 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766497 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766501 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766505 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766508 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.786052 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766512 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766516 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766519 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766523 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766527 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766531 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766534 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766539 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766543 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766546 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766551 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766555 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766560 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766563 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766567 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766571 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766574 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766578 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766582 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766585 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:02.786699 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766589 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766594 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766597 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766601 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766605 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766609 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766613 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766617 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766621 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766625 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766642 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766646 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.766650 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767681 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767691 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767696 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767701 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767705 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767709 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767714 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:02.787296 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767718 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767722 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767726 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767730 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767734 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767739 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767743 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767747 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767751 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767755 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767759 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767763 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767766 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767770 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767774 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767778 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767795 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767801 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767805 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767810 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:02.787902 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767813 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767819 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767825 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767831 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767835 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767841 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767847 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767851 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767855 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767859 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767863 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767867 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767870 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767875 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767879 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767882 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767886 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767890 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767894 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:02.788593 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767900 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767904 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767908 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767912 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767916 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767919 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767923 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767927 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767931 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767935 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767938 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767943 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767947 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767951 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767956 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767960 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767963 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767967 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767971 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767974 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:02.789280 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767978 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767981 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767985 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767988 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767992 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767996 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.767999 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768003 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768007 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768010 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768014 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768018 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768023 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768027 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768031 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768035 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768038 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768042 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768046 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768050 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.790022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768054 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768058 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768061 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768065 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768068 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768072 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768076 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768080 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768085 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768088 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768092 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768096 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768100 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768645 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768651 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768656 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768660 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768666 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768671 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768675 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:02.791022 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768679 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768683 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768689 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768694 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768700 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768704 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768708 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768712 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768716 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768720 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768724 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768727 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768731 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768735 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768739 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768742 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768747 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768750 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768754 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768758 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.917302 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768762 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:02.839250 ip-10-0-128-249 systemd[1]: Started Kubernetes Kubelet. Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768766 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768770 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768774 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768778 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768783 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768787 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768790 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768794 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768799 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768803 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768807 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768811 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768815 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768819 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768823 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768827 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768832 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768836 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768839 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.919059 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768843 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768847 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768850 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768854 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768858 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768862 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768866 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768869 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768873 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768877 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768881 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768885 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768889 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768893 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768898 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768904 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768909 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768913 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768917 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:02.920292 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768921 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768926 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768931 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768935 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768939 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768943 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768958 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768962 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768966 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768970 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768974 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768978 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768982 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768986 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768990 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768993 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.768997 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769001 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769005 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769008 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:02.921362 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769012 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769016 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769019 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769023 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769027 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769031 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769034 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769041 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769047 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769051 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769054 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769058 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.769062 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769149 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769171 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769179 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769185 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769191 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769196 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769203 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 13:10:02.922252 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769209 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769213 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769217 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769222 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769228 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769232 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769237 2573 flags.go:64] FLAG: --cgroup-root="" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769241 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769245 2573 flags.go:64] FLAG: --client-ca-file="" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769249 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769253 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769259 2573 flags.go:64] FLAG: --cluster-domain="" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769263 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769268 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769272 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769277 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769283 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769287 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769291 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769299 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769304 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769308 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769312 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769317 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 13:10:02.923348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769322 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769327 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769333 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769338 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769342 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769347 2573 flags.go:64] FLAG: --enable-server="true" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769351 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769357 2573 flags.go:64] FLAG: --event-burst="100" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769361 2573 flags.go:64] FLAG: --event-qps="50" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769366 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769370 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769374 2573 flags.go:64] FLAG: --eviction-hard="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769380 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769385 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769389 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769394 2573 flags.go:64] FLAG: --eviction-soft="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769398 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769403 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769407 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769411 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769415 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769419 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769424 2573 flags.go:64] FLAG: --feature-gates="" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769429 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769434 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 13:10:02.924443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769438 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769442 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769448 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769452 2573 flags.go:64] FLAG: --help="false" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769457 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-249.ec2.internal" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769461 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769466 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769470 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769474 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769481 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769485 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769489 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769494 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769498 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769502 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769506 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769511 2573 flags.go:64] FLAG: --kube-reserved="" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769516 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769520 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769525 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769529 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769534 2573 flags.go:64] FLAG: --lock-file="" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769538 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769542 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 13:10:02.925960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769546 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769554 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769559 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769563 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769567 2573 flags.go:64] FLAG: --logging-format="text" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769572 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769576 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769581 2573 flags.go:64] FLAG: --manifest-url="" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769585 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769591 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769597 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769602 2573 flags.go:64] FLAG: --max-pods="110" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769607 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769611 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769615 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769620 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769625 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769647 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769652 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769664 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769669 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769674 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769678 2573 flags.go:64] FLAG: --pod-cidr="" Jul 10 13:10:02.927062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769682 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769691 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769695 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769700 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769704 2573 flags.go:64] FLAG: --port="10250" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769709 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769713 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09410cb9a460a1706" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769719 2573 flags.go:64] FLAG: --qos-reserved="" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769723 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769727 2573 flags.go:64] FLAG: --register-node="true" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769732 2573 flags.go:64] FLAG: --register-with-taints="" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769737 2573 flags.go:64] FLAG: --registry-burst="10" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769741 2573 flags.go:64] FLAG: --registry-qps="5" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769745 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769749 2573 flags.go:64] FLAG: --reserved-memory="" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769754 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769759 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769763 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769767 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769773 2573 flags.go:64] FLAG: --runonce="false" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769777 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769782 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769786 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769790 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769794 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769798 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 13:10:02.928183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769803 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769808 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769812 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769817 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769821 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769824 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769829 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769833 2573 flags.go:64] FLAG: --system-cgroups="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769837 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769846 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769850 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769854 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769861 2573 flags.go:64] FLAG: --tls-min-version="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769865 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769870 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769875 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769879 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769884 2573 flags.go:64] FLAG: --v="2" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769889 2573 flags.go:64] FLAG: --version="false" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769894 2573 flags.go:64] FLAG: --vmodule="" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769900 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.769904 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770120 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770126 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.929339 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770130 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770138 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770143 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770148 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770152 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770156 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770161 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770165 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770169 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770175 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770178 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770182 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770186 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770190 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770194 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770198 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770202 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770205 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770210 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:02.930441 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770213 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770217 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770221 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770225 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770238 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770243 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770246 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770250 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770254 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770257 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770261 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770265 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770269 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770273 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770278 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770282 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770286 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770290 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770294 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770298 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:02.933486 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770301 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770305 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770311 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770316 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770321 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770325 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770330 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770334 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770338 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770342 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770347 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770351 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770355 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770359 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770364 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770368 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770372 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770383 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770388 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770392 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:02.934504 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770396 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770400 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770404 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770408 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770412 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770416 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770421 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770425 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770429 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770432 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770436 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770441 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770446 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770451 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770456 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770460 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770464 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770467 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770471 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.937983 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770475 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770479 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770483 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770486 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770490 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770494 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770498 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770502 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770506 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770510 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770514 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770525 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770529 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770532 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770536 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770540 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770544 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770547 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:02.939351 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.770551 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.771638 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.780750 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.780767 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.780787 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.780799 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780884 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780892 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780897 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780901 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780906 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780910 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780914 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780919 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780924 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:02.957927 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780928 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780932 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780936 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780940 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780944 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780957 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780961 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780965 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780969 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780973 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780977 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780981 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780984 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780991 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.780996 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781001 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781008 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781012 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781017 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:02.958380 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781021 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781034 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781038 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781046 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781050 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781054 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781058 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781062 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781065 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781070 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781074 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781078 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781082 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781086 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781090 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781094 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781098 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781102 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781108 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.959416 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781113 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781118 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781123 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781128 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781132 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781136 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781140 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781144 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781150 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781154 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781158 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781162 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781166 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781170 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781184 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781188 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781192 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781196 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781200 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781204 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:02.960226 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781208 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781212 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781216 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781219 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781223 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781227 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781231 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781234 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781238 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781241 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781245 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781249 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781253 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781256 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781260 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781264 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781268 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781271 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781275 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781280 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.961194 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781285 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781290 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781293 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781298 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781301 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781305 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781309 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781325 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781329 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781332 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781337 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.781344 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781562 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781570 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781575 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:02.962073 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781579 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781583 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781587 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781592 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781596 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781600 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781605 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781610 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781614 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781618 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781622 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781626 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781647 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781651 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781658 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781664 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781669 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781673 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781680 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:02.962930 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781686 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781690 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781694 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781698 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781703 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781709 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781722 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781727 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781732 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781736 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781740 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781744 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781749 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781752 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781756 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781760 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781764 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781767 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781771 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781775 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:03.146776 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781778 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781782 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781786 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781790 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781793 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781799 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781804 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781815 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781819 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781823 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781827 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781833 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781837 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781841 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781844 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781847 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781852 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781857 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781860 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:03.148831 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781872 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781876 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781880 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781883 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781887 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781891 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781895 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781898 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781902 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781905 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781909 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781913 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781916 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781920 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781924 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781927 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781931 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781935 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781939 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781943 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:03.150047 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781946 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781950 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781953 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781957 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781964 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781968 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781971 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781975 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781979 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781983 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781988 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781992 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.781995 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.782006 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.782010 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.782014 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.782018 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:03.152045 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:02.782021 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.782028 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.782191 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.784909 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.786364 2573 server.go:1019] "Starting client certificate rotation" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.786465 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.786532 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.790099 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.795257 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.803648 2573 log.go:25] "Validated CRI v1 runtime API" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.805692 2573 log.go:25] "Validated CRI v1 image API" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.806482 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 13:10:03.152967 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.811177 2573 fs.go:135] Filesystem UUIDs: map[4279b264-3c8b-4e18-9e11-2daeef59ff71:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e941827c-75b2-4677-8e73-774ab65ba6e8:/dev/nvme0n1p4] Jul 10 13:10:03.153681 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.811200 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 13:10:03.153681 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.815147 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.816282 2573 manager.go:216] Machine: {Timestamp:2026-07-10 13:10:02.814992082 +0000 UTC m=+0.347594373 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087337 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20911615d8118f864c6494a9927f9d SystemUUID:ec209116-15d8-118f-864c-6494a9927f9d BootID:2225a189-9cd2-4f39-9f96-d25dbb9909ea Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9f:22:6c:b9:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9f:22:6c:b9:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:37:0c:43:53:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.816385 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.816462 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.818777 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.818805 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.819556 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.819566 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.819574 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.820823 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.821675 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824309 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824423 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824451 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824461 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824477 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824486 2573 kubelet.go:398] "Adding apiserver pod source" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824494 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 13:10:03.155522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824939 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.824967 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.827964 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.829478 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.829506 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830202 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830220 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830227 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830234 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830242 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830248 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830320 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830325 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830332 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830337 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830351 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.830368 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.831734 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.831750 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.834995 2573 server.go:1268] "Started kubelet" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.835107 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.835183 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.835137 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.840009 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.840016 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.840991 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.847475 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.847697 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.848500 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.850217 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-249.ec2.internal.18c0efb29bbfb5e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-249.ec2.internal,UID:ip-10-0-128-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-249.ec2.internal,},FirstTimestamp:2026-07-10 13:10:02.83496599 +0000 UTC m=+0.367568282,LastTimestamp:2026-07-10 13:10:02.83496599 +0000 UTC m=+0.367568282,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-249.ec2.internal,}" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.851478 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.851487 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.851537 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852090 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852092 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852122 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852227 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852232 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852443 2573 factory.go:55] Registering systemd factory Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852458 2573 factory.go:223] Registration of the systemd container factory successfully Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852626 2573 factory.go:153] Registering CRI-O factory Jul 10 13:10:03.157370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852659 2573 factory.go:223] Registration of the crio container factory successfully Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852707 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852732 2573 factory.go:103] Registering Raw factory Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.852749 2573 manager.go:1196] Started watching for new ooms in manager Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.854299 2573 manager.go:349] Starting recovery of all containers Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.855798 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.855812 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.869687 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cr2rk" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.871617 2573 manager.go:354] Recovery completed Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.875496 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cr2rk" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.878126 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.879933 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.879966 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.879982 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.880452 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.880463 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.880474 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.880479 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881230 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881255 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881268 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881842 2573 policy_none.go:49] "None policy: Start" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881858 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.881871 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.882078 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-249.ec2.internal.18c0efb29e6e1d96 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-249.ec2.internal,UID:ip-10-0-128-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-249.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-249.ec2.internal,},FirstTimestamp:2026-07-10 13:10:02.87995023 +0000 UTC m=+0.412552523,LastTimestamp:2026-07-10 13:10:02.87995023 +0000 UTC m=+0.412552523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-249.ec2.internal,}" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.882865 2573 policy_none.go:47] "Start" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.951737 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959279 2573 manager.go:337] "Starting Device Plugin manager" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.959486 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959500 2573 server.go:87] "Starting device plugin registration server" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959784 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959799 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959877 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959972 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.959982 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.960108 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.961401 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.961451 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.962228 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 13:10:03.159273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.962269 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.962292 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:02.962331 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:02.964850 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.060597 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.061515 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.061545 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.061556 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.061578 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.061972 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-249.ec2.internal\" not found" node="ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.063126 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal"] Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.063178 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.063860 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.063882 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.063891 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065189 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065392 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065868 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065885 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.065898 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.066938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.066956 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067068 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067081 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067089 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067788 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067804 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.067814 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.072061 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.072076 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-249.ec2.internal\": node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.091183 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-249.ec2.internal\" not found" node="ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.095070 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-249.ec2.internal\" not found" node="ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.101594 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.155861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6184f72f327664492ecaec42e6fc927e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-249.ec2.internal\" (UID: \"6184f72f327664492ecaec42e6fc927e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.162295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.155917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.163954 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.155954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.201736 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6184f72f327664492ecaec42e6fc927e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-249.ec2.internal\" (UID: \"6184f72f327664492ecaec42e6fc927e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6184f72f327664492ecaec42e6fc927e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-249.ec2.internal\" (UID: \"6184f72f327664492ecaec42e6fc927e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.256542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d30e163e2fcc5fe929b201de2320d3f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal\" (UID: \"d30e163e2fcc5fe929b201de2320d3f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.352660 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.302760 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.489739 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.396417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.489739 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.397592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:03.489739 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.403594 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.644337 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.506428 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.644337 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.607120 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.863138 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.710263 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.863138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.785397 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 13:10:03.863138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.785575 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 13:10:03.863138 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.812601 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:03.863138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.854874 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:04.051930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.879545 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 13:05:03 +0000 UTC" deadline="2028-03-13 04:10:14.966644063 +0000 UTC" Jul 10 13:10:04.051930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:03.879581 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14679h0m11.087066676s" Jul 10 13:10:04.051930 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:03.915403 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:04.051930 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.016852 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-249.ec2.internal\" not found" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.090855 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.152091 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.165476 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.166701 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.174524 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:10:04.226211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.217515 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:04.377239 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:04.369713 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6184f72f327664492ecaec42e6fc927e.slice/crio-3260016cb5a8e14616cb9f79586d7a6c75022b99e55f79150d39a8b102b2164e WatchSource:0}: Error finding container 3260016cb5a8e14616cb9f79586d7a6c75022b99e55f79150d39a8b102b2164e: Status 404 returned error can't find the container with id 3260016cb5a8e14616cb9f79586d7a6c75022b99e55f79150d39a8b102b2164e Jul 10 13:10:04.377239 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:04.369991 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30e163e2fcc5fe929b201de2320d3f7.slice/crio-33666fe3116e26451a0588d95143bb91bd6b5ad98a45b408b218d95582b4dee5 WatchSource:0}: Error finding container 33666fe3116e26451a0588d95143bb91bd6b5ad98a45b408b218d95582b4dee5: Status 404 returned error can't find the container with id 33666fe3116e26451a0588d95143bb91bd6b5ad98a45b408b218d95582b4dee5 Jul 10 13:10:04.377239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.375505 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:10:04.825859 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.825771 2573 apiserver.go:52] "Watching apiserver" Jul 10 13:10:04.835306 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.835272 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 13:10:04.837212 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.837185 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4","openshift-cluster-node-tuning-operator/tuned-rhgjg","openshift-image-registry/node-ca-4bp48","openshift-multus/multus-vkgbg","openshift-network-operator/iptables-alerter-4mdqh","openshift-ovn-kubernetes/ovnkube-node-qjbtb","kube-system/global-pull-secret-syncer-28kx4","kube-system/konnectivity-agent-xt2hl","kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal","openshift-dns/node-resolver-6zb9b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal","openshift-multus/multus-additional-cni-plugins-5qsqd","openshift-multus/network-metrics-daemon-t95jr","openshift-network-diagnostics/network-check-target-nmbs2"] Jul 10 13:10:04.839736 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.839718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.839827 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.839801 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:04.841037 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.841017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.841134 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.841104 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:04.842380 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.842361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.844218 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.844193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.844922 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.844884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.845011 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.844939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 13:10:04.845083 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.844884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.845083 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.845058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m8lbz\"" Jul 10 13:10:04.845878 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.845855 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.846403 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.846382 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.846653 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.846602 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 13:10:04.846789 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.846604 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.846856 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.846815 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lwwxw\"" Jul 10 13:10:04.846943 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.846926 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 13:10:04.847919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.847900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.848504 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.848481 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.848691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.848659 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.848857 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.848840 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 13:10:04.849214 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.849195 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z79gv\"" Jul 10 13:10:04.849463 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.849446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:04.850321 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.850305 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 13:10:04.850528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.850516 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.850716 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.850672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:04.851540 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.851525 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 13:10:04.853449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.852080 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gd88g\"" Jul 10 13:10:04.853449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.852296 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.853449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.852403 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 13:10:04.853449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.852556 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 13:10:04.853449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.852603 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 13:10:04.854104 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.853922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.855912 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.855895 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.856909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.856893 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 13:10:04.857014 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.856916 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.857014 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.856917 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.857387 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.857368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:04.857473 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.857398 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pnd2n\"" Jul 10 13:10:04.858026 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.858008 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.858797 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.858784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-chfvv\"" Jul 10 13:10:04.859036 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.859020 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.859299 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.859283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:04.861172 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.861153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.867355 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-node-log\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-env-overrides\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-etc-kubernetes\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-etc-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-device-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-host-slash\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-netd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cnibin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-daemon-config\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-kubelet\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-log-socket\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrn8\" (UniqueName: \"kubernetes.io/projected/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-kube-api-access-vgrn8\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-iptables-alerter-script\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-bin\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hwv\" (UniqueName: \"kubernetes.io/projected/cba75e71-6c03-4086-a4f6-e2183cbd64a3-kube-api-access-h6hwv\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.867891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-os-release\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-multus\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-slash\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-config\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovn-node-metrics-cert\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-sys-fs\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-hostroot\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-systemd-units\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.867995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cni-binary-copy\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c526be69-1738-4d64-8989-bb9c151b0ce6-serviceca\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-conf-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tx5b\" (UniqueName: \"kubernetes.io/projected/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-kube-api-access-7tx5b\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-ovn\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrwr\" (UniqueName: \"kubernetes.io/projected/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kube-api-access-jfrwr\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.868644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c526be69-1738-4d64-8989-bb9c151b0ce6-host\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-var-lib-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-registration-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/c526be69-1738-4d64-8989-bb9c151b0ce6-kube-api-access-j489t\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-bin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-systemd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-socket-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868392 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-k8s-cni-cncf-io\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-netns\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhxx\" (UniqueName: \"kubernetes.io/projected/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-kube-api-access-srhxx\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-system-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-socket-dir-parent\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-kubelet-config\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.869427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-dbus\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-netns\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-script-lib\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-kubelet\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868747 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9lm7b\"" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868719 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-multus-certs\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.868831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.869909 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.869964 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v494x\"" Jul 10 13:10:04.870107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.870084 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6ww8h\"" Jul 10 13:10:04.870602 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.870271 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 13:10:04.874433 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.874415 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:04.884943 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.884925 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:10:04.903510 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.903491 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnrjs" Jul 10 13:10:04.915111 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.915092 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnrjs" Jul 10 13:10:04.953009 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.952989 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 13:10:04.968979 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.968952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-var-lib-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.968987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-registration-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/c526be69-1738-4d64-8989-bb9c151b0ce6-kube-api-access-j489t\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-bin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-var-lib-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-tuned\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.969131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-bin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-systemd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969187 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-systemd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-socket-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-k8s-cni-cncf-io\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-registration-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-netns\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-sys\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-k8s-cni-cncf-io\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhxx\" (UniqueName: \"kubernetes.io/projected/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-kube-api-access-srhxx\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-system-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-socket-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-socket-dir-parent\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.969391 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-kubelet-config\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.969549 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.969443 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:05.469424392 +0000 UTC m=+3.002026672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-dbus\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-kubelet-config\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-agent-certs\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-netns\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-script-lib\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-kubelet\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-multus-certs\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-host\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-socket-dir-parent\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-run\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-system-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-node-log\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-env-overrides\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-etc-kubernetes\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969789 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-node-log\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47874066-db3d-4c40-9c31-e4ee15f49de4-tmp-dir\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cnibin\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-etc-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-device-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969910 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-etc-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-lib-modules\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-host-slash\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" event={"ID":"6184f72f327664492ecaec42e6fc927e","Type":"ContainerStarted","Data":"3260016cb5a8e14616cb9f79586d7a6c75022b99e55f79150d39a8b102b2164e"} Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.969990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-netd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-etc-kubernetes\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cnibin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.970832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cnibin\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-daemon-config\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-netd\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-netns\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970201 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-host-slash\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-conf\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-var-lib-kubelet\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-kubelet\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-log-socket\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrn8\" (UniqueName: \"kubernetes.io/projected/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-kube-api-access-vgrn8\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysconfig\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36ab361d-d9c8-4d97-bf03-4393b79c32f8-dbus\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-device-dir\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-run-netns\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-env-overrides\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.971668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-systemd\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-kubelet\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-tmp\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-log-socket\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-iptables-alerter-script\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-bin\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hwv\" (UniqueName: \"kubernetes.io/projected/cba75e71-6c03-4086-a4f6-e2183cbd64a3-kube-api-access-h6hwv\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-os-release\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-daemon-config\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-multus\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-kubernetes\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-system-cni-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970700 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-run-multus-certs\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-slash\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-cni-bin\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-kubelet\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.972544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-config\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970789 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-host-var-lib-cni-multus\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6m8\" (UniqueName: \"kubernetes.io/projected/0cba6022-af14-4e4d-826f-ec6c02510e39-kube-api-access-sj6m8\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-host-slash\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-os-release\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovn-node-metrics-cert\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-sys-fs\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.970988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-openvswitch\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-hostroot\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-sys-fs\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-script-lib\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-cni-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971300 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovnkube-config\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-modprobe-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971352 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-hostroot\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-konnectivity-ca\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:04.973080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-iptables-alerter-script\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-systemd-units\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cni-binary-copy\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlxnc\" (UniqueName: \"kubernetes.io/projected/47874066-db3d-4c40-9c31-e4ee15f49de4-kube-api-access-jlxnc\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-systemd-units\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c526be69-1738-4d64-8989-bb9c151b0ce6-serviceca\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-conf-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" event={"ID":"d30e163e2fcc5fe929b201de2320d3f7","Type":"ContainerStarted","Data":"33666fe3116e26451a0588d95143bb91bd6b5ad98a45b408b218d95582b4dee5"} Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971648 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.971670 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tx5b\" (UniqueName: \"kubernetes.io/projected/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-kube-api-access-7tx5b\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.971709 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:05.471698785 +0000 UTC m=+3.004301068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-os-release\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtjs\" (UniqueName: \"kubernetes.io/projected/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-kube-api-access-qjtjs\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-multus-conf-dir\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.973600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-ovn\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrwr\" (UniqueName: \"kubernetes.io/projected/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kube-api-access-jfrwr\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.971927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba75e71-6c03-4086-a4f6-e2183cbd64a3-run-ovn\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.972043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c526be69-1738-4d64-8989-bb9c151b0ce6-host\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.972098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47874066-db3d-4c40-9c31-e4ee15f49de4-hosts-file\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.972114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c526be69-1738-4d64-8989-bb9c151b0ce6-serviceca\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.972135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c526be69-1738-4d64-8989-bb9c151b0ce6-host\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.974136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.972841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-cni-binary-copy\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:04.975069 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.975048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba75e71-6c03-4086-a4f6-e2183cbd64a3-ovn-node-metrics-cert\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.984759 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.984739 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:04.984759 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.984759 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:04.984878 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.984769 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:04.984878 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:04.984822 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:05.484810772 +0000 UTC m=+3.017413065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:04.988026 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.988007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hwv\" (UniqueName: \"kubernetes.io/projected/cba75e71-6c03-4086-a4f6-e2183cbd64a3-kube-api-access-h6hwv\") pod \"ovnkube-node-qjbtb\" (UID: \"cba75e71-6c03-4086-a4f6-e2183cbd64a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:04.988132 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.988073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrn8\" (UniqueName: \"kubernetes.io/projected/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-kube-api-access-vgrn8\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:04.989196 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.989164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j489t\" (UniqueName: \"kubernetes.io/projected/c526be69-1738-4d64-8989-bb9c151b0ce6-kube-api-access-j489t\") pod \"node-ca-4bp48\" (UID: \"c526be69-1738-4d64-8989-bb9c151b0ce6\") " pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:04.989397 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.989380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrwr\" (UniqueName: \"kubernetes.io/projected/fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966-kube-api-access-jfrwr\") pod \"aws-ebs-csi-driver-node-v9wv4\" (UID: \"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:04.989445 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.989427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhxx\" (UniqueName: \"kubernetes.io/projected/fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb-kube-api-access-srhxx\") pod \"iptables-alerter-4mdqh\" (UID: \"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb\") " pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:04.989479 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:04.989427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tx5b\" (UniqueName: \"kubernetes.io/projected/2067e9e7-8b4f-4523-9e4f-c409f70ac1a1-kube-api-access-7tx5b\") pod \"multus-vkgbg\" (UID: \"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1\") " pod="openshift-multus/multus-vkgbg" Jul 10 13:10:05.073383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-conf\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-var-lib-kubelet\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysconfig\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-systemd\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-tmp\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-var-lib-kubelet\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073620 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysconfig\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073668 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-systemd\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-sysctl-conf\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-kubernetes\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-system-cni-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073775 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-kubernetes\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6m8\" (UniqueName: \"kubernetes.io/projected/0cba6022-af14-4e4d-826f-ec6c02510e39-kube-api-access-sj6m8\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-modprobe-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-konnectivity-ca\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:05.073874 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxnc\" (UniqueName: \"kubernetes.io/projected/47874066-db3d-4c40-9c31-e4ee15f49de4-kube-api-access-jlxnc\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-os-release\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.073967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtjs\" (UniqueName: \"kubernetes.io/projected/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-kube-api-access-qjtjs\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-system-cni-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-os-release\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-modprobe-d\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47874066-db3d-4c40-9c31-e4ee15f49de4-hosts-file\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-tuned\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-sys\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-agent-certs\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-host\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-run\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47874066-db3d-4c40-9c31-e4ee15f49de4-tmp-dir\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cnibin\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-lib-modules\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-konnectivity-ca\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:05.074852 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.074851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47874066-db3d-4c40-9c31-e4ee15f49de4-hosts-file\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.075333 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.075406 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-run\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.075458 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.075515 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-sys\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.075697 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47874066-db3d-4c40-9c31-e4ee15f49de4-tmp-dir\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.075801 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075768 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-cnibin\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.075935 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-lib-modules\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.076005 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.075946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba6022-af14-4e4d-826f-ec6c02510e39-host\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.076351 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.076327 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-tmp\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.077699 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.077551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cba6022-af14-4e4d-826f-ec6c02510e39-etc-tuned\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.078383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.078150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b-agent-certs\") pod \"konnectivity-agent-xt2hl\" (UID: \"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b\") " pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:05.082906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.082404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6m8\" (UniqueName: \"kubernetes.io/projected/0cba6022-af14-4e4d-826f-ec6c02510e39-kube-api-access-sj6m8\") pod \"tuned-rhgjg\" (UID: \"0cba6022-af14-4e4d-826f-ec6c02510e39\") " pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.082906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.082814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtjs\" (UniqueName: \"kubernetes.io/projected/82bbaed0-7855-4587-b7c4-2d4b15ce23ea-kube-api-access-qjtjs\") pod \"multus-additional-cni-plugins-5qsqd\" (UID: \"82bbaed0-7855-4587-b7c4-2d4b15ce23ea\") " pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.082906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.082855 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxnc\" (UniqueName: \"kubernetes.io/projected/47874066-db3d-4c40-9c31-e4ee15f49de4-kube-api-access-jlxnc\") pod \"node-resolver-6zb9b\" (UID: \"47874066-db3d-4c40-9c31-e4ee15f49de4\") " pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.155594 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.155555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bp48" Jul 10 13:10:05.160633 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.160611 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkgbg" Jul 10 13:10:05.168539 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.168454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mdqh" Jul 10 13:10:05.173065 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.173019 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2067e9e7_8b4f_4523_9e4f_c409f70ac1a1.slice/crio-0a5c6d301d164e9ea35cd60e82331b69e77b2f5f774c90753ee4f6ac45dc375b WatchSource:0}: Error finding container 0a5c6d301d164e9ea35cd60e82331b69e77b2f5f774c90753ee4f6ac45dc375b: Status 404 returned error can't find the container with id 0a5c6d301d164e9ea35cd60e82331b69e77b2f5f774c90753ee4f6ac45dc375b Jul 10 13:10:05.173630 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.173606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:05.180054 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.180027 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" Jul 10 13:10:05.182346 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.182314 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe0950f_10ff_4ef6_8cf6_a4a10d0afacb.slice/crio-9202ee0113f6ebe269127e4830bff8b8cb765ac621071626e5953579a8592eb8 WatchSource:0}: Error finding container 9202ee0113f6ebe269127e4830bff8b8cb765ac621071626e5953579a8592eb8: Status 404 returned error can't find the container with id 9202ee0113f6ebe269127e4830bff8b8cb765ac621071626e5953579a8592eb8 Jul 10 13:10:05.185445 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.185424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" Jul 10 13:10:05.187181 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.187154 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba75e71_6c03_4086_a4f6_e2183cbd64a3.slice/crio-4571ae78f90bfa4a2189bd80e54e5c2f5d59afa22df66fab81c644a8e8cd047d WatchSource:0}: Error finding container 4571ae78f90bfa4a2189bd80e54e5c2f5d59afa22df66fab81c644a8e8cd047d: Status 404 returned error can't find the container with id 4571ae78f90bfa4a2189bd80e54e5c2f5d59afa22df66fab81c644a8e8cd047d Jul 10 13:10:05.191300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.191111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zb9b" Jul 10 13:10:05.192191 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.192151 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6efa2c_a9ad_4dd8_9f05_9f34ca6be966.slice/crio-478d65d6d0cd9d478455742d20734e1a53a7cd5b4743136f1d39cb17e068d42e WatchSource:0}: Error finding container 478d65d6d0cd9d478455742d20734e1a53a7cd5b4743136f1d39cb17e068d42e: Status 404 returned error can't find the container with id 478d65d6d0cd9d478455742d20734e1a53a7cd5b4743136f1d39cb17e068d42e Jul 10 13:10:05.195534 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.195507 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cba6022_af14_4e4d_826f_ec6c02510e39.slice/crio-f2e75d1420b3da1cb20b50b36f4859d98e27f8a479bc9550b13dc2da0b090a15 WatchSource:0}: Error finding container f2e75d1420b3da1cb20b50b36f4859d98e27f8a479bc9550b13dc2da0b090a15: Status 404 returned error can't find the container with id f2e75d1420b3da1cb20b50b36f4859d98e27f8a479bc9550b13dc2da0b090a15 Jul 10 13:10:05.198439 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.198423 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:05.202605 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.202568 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" Jul 10 13:10:05.203973 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.203891 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47874066_db3d_4c40_9c31_e4ee15f49de4.slice/crio-3e5a0c2d7444f4101cbf335c0eeed3f0c78f8fcc197280237984eaa67b7a80c9 WatchSource:0}: Error finding container 3e5a0c2d7444f4101cbf335c0eeed3f0c78f8fcc197280237984eaa67b7a80c9: Status 404 returned error can't find the container with id 3e5a0c2d7444f4101cbf335c0eeed3f0c78f8fcc197280237984eaa67b7a80c9 Jul 10 13:10:05.215660 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.214780 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e7b8a3_80e5_4c3a_b613_ae8f57e8975b.slice/crio-a048ff82478f1910ef58a0f74fdff9825c01357e28c3c7cc3571fbc186bc4f58 WatchSource:0}: Error finding container a048ff82478f1910ef58a0f74fdff9825c01357e28c3c7cc3571fbc186bc4f58: Status 404 returned error can't find the container with id a048ff82478f1910ef58a0f74fdff9825c01357e28c3c7cc3571fbc186bc4f58 Jul 10 13:10:05.218689 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:05.218605 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bbaed0_7855_4587_b7c4_2d4b15ce23ea.slice/crio-b2514a167ca45672ff592cd9bcedfd6fe8d30253c8b8323bca555b022a14a149 WatchSource:0}: Error finding container b2514a167ca45672ff592cd9bcedfd6fe8d30253c8b8323bca555b022a14a149: Status 404 returned error can't find the container with id b2514a167ca45672ff592cd9bcedfd6fe8d30253c8b8323bca555b022a14a149 Jul 10 13:10:05.478611 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.478517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:05.478611 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.478569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:05.478827 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.478686 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:05.478827 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.478713 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:05.478827 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.478742 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:06.478724095 +0000 UTC m=+4.011326376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:05.478827 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.478780 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:06.478765924 +0000 UTC m=+4.011368204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:05.579582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.579547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:05.579788 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.579764 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:05.579854 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.579795 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:05.579854 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.579810 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:05.579942 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.579869 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:06.579854867 +0000 UTC m=+4.112457144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:05.917789 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.917668 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:05:05 +0000 UTC" deadline="2028-03-06 10:29:24.235954332 +0000 UTC" Jul 10 13:10:05.917789 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.917735 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14517h19m18.318223622s" Jul 10 13:10:05.967623 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.967068 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:05.967623 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.967086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:05.967623 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.967222 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:05.967623 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:05.967375 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:05.980060 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.979979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerStarted","Data":"b2514a167ca45672ff592cd9bcedfd6fe8d30253c8b8323bca555b022a14a149"} Jul 10 13:10:05.981885 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.981850 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" event={"ID":"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966","Type":"ContainerStarted","Data":"478d65d6d0cd9d478455742d20734e1a53a7cd5b4743136f1d39cb17e068d42e"} Jul 10 13:10:05.985126 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.985095 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkgbg" event={"ID":"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1","Type":"ContainerStarted","Data":"0a5c6d301d164e9ea35cd60e82331b69e77b2f5f774c90753ee4f6ac45dc375b"} Jul 10 13:10:05.989177 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.989115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bp48" event={"ID":"c526be69-1738-4d64-8989-bb9c151b0ce6","Type":"ContainerStarted","Data":"8127bcefee0c46481d99b0e7d78bc3019fcee334e86268acf980eea409aaf343"} Jul 10 13:10:05.994823 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.994792 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xt2hl" event={"ID":"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b","Type":"ContainerStarted","Data":"a048ff82478f1910ef58a0f74fdff9825c01357e28c3c7cc3571fbc186bc4f58"} Jul 10 13:10:05.999157 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:05.999064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zb9b" event={"ID":"47874066-db3d-4c40-9c31-e4ee15f49de4","Type":"ContainerStarted","Data":"3e5a0c2d7444f4101cbf335c0eeed3f0c78f8fcc197280237984eaa67b7a80c9"} Jul 10 13:10:06.004159 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.004123 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" event={"ID":"0cba6022-af14-4e4d-826f-ec6c02510e39","Type":"ContainerStarted","Data":"f2e75d1420b3da1cb20b50b36f4859d98e27f8a479bc9550b13dc2da0b090a15"} Jul 10 13:10:06.007908 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.007878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"4571ae78f90bfa4a2189bd80e54e5c2f5d59afa22df66fab81c644a8e8cd047d"} Jul 10 13:10:06.010180 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.010147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mdqh" event={"ID":"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb","Type":"ContainerStarted","Data":"9202ee0113f6ebe269127e4830bff8b8cb765ac621071626e5953579a8592eb8"} Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.487633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.487714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.487854 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.487921 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:08.487901076 +0000 UTC m=+6.020503361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.488323 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:06.488410 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.488375 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:08.488359281 +0000 UTC m=+6.020961575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:06.588506 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.588465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:06.588730 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.588689 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:06.588730 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.588716 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:06.588730 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.588731 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:06.588919 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.588795 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:08.588773174 +0000 UTC m=+6.121375459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:06.918238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.918150 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:05:05 +0000 UTC" deadline="2027-12-20 20:15:15.771478203 +0000 UTC" Jul 10 13:10:06.918238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.918189 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12679h5m8.853293303s" Jul 10 13:10:06.966613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:06.966019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:06.966613 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:06.966173 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:07.962716 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:07.962647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:07.963165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:07.962771 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:07.963301 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:07.963239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:07.963356 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:07.963340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:08.504328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:08.504393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.504544 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.504607 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:12.504588006 +0000 UTC m=+10.037190306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.505012 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:08.505095 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.505061 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:12.505045462 +0000 UTC m=+10.037647743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:08.605138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:08.605098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:08.605323 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.605306 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:08.605391 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.605333 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:08.605391 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.605349 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:08.605497 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.605424 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:12.605404363 +0000 UTC m=+10.138006640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:08.963129 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:08.962951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:08.963129 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:08.963101 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:09.962996 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:09.962949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:09.963194 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:09.962972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:09.963194 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:09.963095 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:09.963194 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:09.963148 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:10.963709 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:10.963615 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:10.964143 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:10.963782 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:11.963417 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:11.963387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:11.963595 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:11.963386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:11.963595 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:11.963518 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:11.963727 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:11.963646 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:12.537584 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:12.537373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:12.537584 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:12.537441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:12.537584 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.537554 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:12.537828 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.537566 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:12.537828 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.537667 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:20.537645788 +0000 UTC m=+18.070248082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:12.537828 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.537707 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:20.537692875 +0000 UTC m=+18.070295155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:12.638081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:12.638039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:12.638251 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.638237 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:12.638337 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.638260 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:12.638337 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.638273 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:12.638337 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.638336 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:20.638317127 +0000 UTC m=+18.170919426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:12.966429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:12.965778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:12.966429 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:12.965915 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:13.963268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:13.963160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:13.963268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:13.963212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:13.963497 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:13.963295 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:13.963710 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:13.963665 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:14.963591 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:14.963559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:14.964106 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:14.963715 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:15.963437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:15.963248 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:15.963890 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:15.963543 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:15.963890 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:15.963291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:15.963890 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:15.963737 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:16.044029 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:16.043958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" event={"ID":"0cba6022-af14-4e4d-826f-ec6c02510e39","Type":"ContainerStarted","Data":"92f069f9c3a347c5bada7ea4a9569da680eb56b0014f9204501918fd73a7eb4d"} Jul 10 13:10:16.045600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:16.045578 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" event={"ID":"6184f72f327664492ecaec42e6fc927e","Type":"ContainerStarted","Data":"3aaa041fd62d3a5aefdb170550f80676bbb0fd86b6efbbd9f353d521663461dc"} Jul 10 13:10:16.074228 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:16.074183 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rhgjg" podStartSLOduration=2.513149554 podStartE2EDuration="13.074162757s" totalImagesPullingTime="10.561013203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:16.062096987 +0000 UTC m=+13.594699287" watchObservedRunningTime="2026-07-10 13:10:16.074162757 +0000 UTC m=+13.606765052" Jul 10 13:10:16.074748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:16.074713 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-249.ec2.internal" podStartSLOduration=12.074703726 podStartE2EDuration="12.074703726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:16.074239477 +0000 UTC m=+13.606841776" watchObservedRunningTime="2026-07-10 13:10:16.074703726 +0000 UTC m=+13.607306068" Jul 10 13:10:16.963310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:16.963227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:16.963515 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:16.963363 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:17.962712 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:17.962476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:17.963204 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:17.962476 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:17.963204 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:17.962801 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:17.963204 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:17.962901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:18.053453 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.053421 2573 generic.go:356] "Generic (PLEG): container finished" podID="d30e163e2fcc5fe929b201de2320d3f7" containerID="89deadbd28e3f1957d7b972e649e6c118d3d6fc0d9eaa5f39778ed638e17debf" exitCode=0 Jul 10 13:10:18.053725 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.053695 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" event={"ID":"d30e163e2fcc5fe929b201de2320d3f7","Type":"ContainerDied","Data":"89deadbd28e3f1957d7b972e649e6c118d3d6fc0d9eaa5f39778ed638e17debf"} Jul 10 13:10:18.055773 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.055746 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="ec5ce9c26ca39801d4dae74f7254f4273d572d23657a3dd1ddd2641472c82d4c" exitCode=0 Jul 10 13:10:18.055860 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.055775 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"ec5ce9c26ca39801d4dae74f7254f4273d572d23657a3dd1ddd2641472c82d4c"} Jul 10 13:10:18.057609 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.057576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" event={"ID":"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966","Type":"ContainerStarted","Data":"3397100da85fd783ed7896b93f5145e47c431ae1ad0bd2914978bad6433dd7a1"} Jul 10 13:10:18.059457 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.059424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bp48" event={"ID":"c526be69-1738-4d64-8989-bb9c151b0ce6","Type":"ContainerStarted","Data":"efa61f96a634efaf3866bdd8b0e08f3ef64c19264a37ffe8bc120a92ac671911"} Jul 10 13:10:18.063456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.063431 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xt2hl" event={"ID":"f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b","Type":"ContainerStarted","Data":"97ffd0107976fdef8b2813a00e0937aed13031292d00d81b0d308412d3683c84"} Jul 10 13:10:18.065907 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.065884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zb9b" event={"ID":"47874066-db3d-4c40-9c31-e4ee15f49de4","Type":"ContainerStarted","Data":"378498b7eaa938a128b8d7ffbe3319678dd11b7b0db7c86bcee16b43b205a1f9"} Jul 10 13:10:18.067381 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.067362 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mdqh" event={"ID":"fbe0950f-10ff-4ef6-8cf6-a4a10d0afacb","Type":"ContainerStarted","Data":"1db7c242e0f65aa9513eff050bc9c5046ff19965c0623551fbb85aff1a10a0d1"} Jul 10 13:10:18.084382 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.084339 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xt2hl" podStartSLOduration=4.635512102 podStartE2EDuration="15.084323246s" totalImagesPullingTime="10.448811144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:18.083838519 +0000 UTC m=+15.616440819" watchObservedRunningTime="2026-07-10 13:10:18.084323246 +0000 UTC m=+15.616925548" Jul 10 13:10:18.107325 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.107272 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4mdqh" podStartSLOduration=4.625633853 podStartE2EDuration="15.107254782s" totalImagesPullingTime="10.481620929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:18.106610817 +0000 UTC m=+15.639213117" watchObservedRunningTime="2026-07-10 13:10:18.107254782 +0000 UTC m=+15.639857086" Jul 10 13:10:18.162485 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.162435 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4bp48" podStartSLOduration=4.667202005 podStartE2EDuration="15.162414818s" totalImagesPullingTime="10.495212813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:18.145253295 +0000 UTC m=+15.677855586" watchObservedRunningTime="2026-07-10 13:10:18.162414818 +0000 UTC m=+15.695017118" Jul 10 13:10:18.271906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.271879 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 13:10:18.290182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.290157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:18.290788 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.290766 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:18.310493 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.310435 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6zb9b" podStartSLOduration=4.7567248410000005 podStartE2EDuration="15.310414152s" totalImagesPullingTime="10.553689311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:18.162033013 +0000 UTC m=+15.694635333" watchObservedRunningTime="2026-07-10 13:10:18.310414152 +0000 UTC m=+15.843016453" Jul 10 13:10:18.963468 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.963327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:18.964179 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:18.963476 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:18.968112 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.968009 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T13:10:18.271902368Z","UUID":"47f68125-1448-48b3-9674-e02bfff56ac5","Handler":null,"Name":"","Endpoint":""} Jul 10 13:10:18.970199 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.970143 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 13:10:18.970199 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:18.970186 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 13:10:19.072004 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.071931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" event={"ID":"d30e163e2fcc5fe929b201de2320d3f7","Type":"ContainerStarted","Data":"9753033175d226a8327227612ccc05c83e5c7ca003a19d997765ef745020a641"} Jul 10 13:10:19.074432 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.074387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" event={"ID":"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966","Type":"ContainerStarted","Data":"3cf87971e5dd156d7bb4dd5a479497c487cdacd7f8315e2b0d5cdf78708810ae"} Jul 10 13:10:19.074674 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.074656 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:19.075188 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.075164 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xt2hl" Jul 10 13:10:19.109130 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.109074 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-249.ec2.internal" podStartSLOduration=15.109052975 podStartE2EDuration="15.109052975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:19.091771861 +0000 UTC m=+16.624374162" watchObservedRunningTime="2026-07-10 13:10:19.109052975 +0000 UTC m=+16.641655278" Jul 10 13:10:19.962754 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.962722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:19.962978 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:19.962722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:19.962978 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:19.962836 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:19.962978 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:19.962934 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:20.602843 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:20.602809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:20.603457 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:20.602862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:20.603457 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.602977 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:20.603457 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.602979 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:20.603457 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.603036 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:36.603019732 +0000 UTC m=+34.135622012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:20.603457 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.603055 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:36.6030456 +0000 UTC m=+34.135647882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:20.703898 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:20.703861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:20.704087 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.704068 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:20.704164 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.704095 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:20.704164 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.704109 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:20.704237 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.704173 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:10:36.704153607 +0000 UTC m=+34.236755909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:20.963386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:20.963302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:20.963515 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:20.963449 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:21.963087 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:21.963054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:21.963685 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:21.963178 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:21.963685 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:21.963223 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:21.963685 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:21.963311 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:22.964413 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:22.964345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:22.964885 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:22.964458 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:23.962765 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:23.962730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:23.962948 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:23.962730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:23.962948 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:23.962860 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:23.963062 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:23.962937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:24.963241 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:24.963209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:24.963784 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:24.963353 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:25.962524 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:25.962488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:25.962832 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:25.962617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:25.962832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:25.962687 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:25.962832 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:25.962809 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:26.963797 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:26.963351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:26.964238 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:26.963926 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:27.095708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095620 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"dde0f75aeac7ee64e306984fe1a48383b8559fdc426cfcb3ffa8234241088e12"} Jul 10 13:10:27.095930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095723 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"1b335def8b2611bfcf38a1630f76a85217da4241884d3313ccf2c412e1c07366"} Jul 10 13:10:27.095930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"52c8ff1d5218ced75e8fab9c26a41948521b6416c157db21095ce49067e4a2ae"} Jul 10 13:10:27.095930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"2474be7a9e509dcf631637604babcff45787290e5e29b8bf86f9d8d3a639b90a"} Jul 10 13:10:27.095930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"266739db4b86dfadb98e03403f8fca8eb6444911b25a164c96648fea64e14a2a"} Jul 10 13:10:27.095930 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.095904 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"e681dd7bd0813916a4ff4da84f065b8ca5f2d1d658fb8f293f6554755b65228a"} Jul 10 13:10:27.097984 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.097956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" event={"ID":"fe6efa2c-a9ad-4dd8-9f05-9f34ca6be966","Type":"ContainerStarted","Data":"1cdd3af6f0d521f65cc341b5709e1bdeaf0e4c4bc84062a1467255e95bacd509"} Jul 10 13:10:27.099730 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.099688 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkgbg" event={"ID":"2067e9e7-8b4f-4523-9e4f-c409f70ac1a1","Type":"ContainerStarted","Data":"ac49c99d7688e41d4adb25483ece9ad2e844b726a8dfc8912f843f8d3062103a"} Jul 10 13:10:27.114049 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.113999 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9wv4" podStartSLOduration=4.436291902 podStartE2EDuration="24.113982867s" totalImagesPullingTime="19.677690965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:27.113963362 +0000 UTC m=+24.646565663" watchObservedRunningTime="2026-07-10 13:10:27.113982867 +0000 UTC m=+24.646585167" Jul 10 13:10:27.133550 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.133511 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vkgbg" podStartSLOduration=2.888246073 podStartE2EDuration="24.133498982s" totalImagesPullingTime="21.245252909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:27.133073537 +0000 UTC m=+24.665675836" watchObservedRunningTime="2026-07-10 13:10:27.133498982 +0000 UTC m=+24.666101280" Jul 10 13:10:27.963182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.963137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:27.963182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:27.963159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:27.963485 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:27.963274 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:27.963485 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:27.963386 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:28.963600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:28.963566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:28.964065 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:28.963719 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:29.962850 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:29.962819 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:29.963071 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:29.962820 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:29.963071 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:29.962946 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:29.963071 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:29.963003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:30.109239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:30.109205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"cb6c65a31e14251866a54e6a838b914fe57678398762f5db6cbbdbe9c2f92c9c"} Jul 10 13:10:30.962944 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:30.962883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:30.963070 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:30.962988 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:31.963035 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:31.962994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:31.963838 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:31.962994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:31.963838 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:31.963115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:31.963838 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:31.963170 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:32.115573 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.115376 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="a8c11fa4087308928cbc8752a90c1bbbdd1cf5d553e97446e290ff1d4dc19c26" exitCode=0 Jul 10 13:10:32.115734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.115460 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"a8c11fa4087308928cbc8752a90c1bbbdd1cf5d553e97446e290ff1d4dc19c26"} Jul 10 13:10:32.119859 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.119837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" event={"ID":"cba75e71-6c03-4086-a4f6-e2183cbd64a3","Type":"ContainerStarted","Data":"748a476c7b78d31f14eee81d8bd4785201e2bb3b4cd61426d382e1375205a5be"} Jul 10 13:10:32.120092 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.120071 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:32.120175 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.120133 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:32.120220 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.120177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:32.137312 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.137279 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:32.137448 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.137434 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:10:32.170647 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.170581 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" podStartSLOduration=7.938779062 podStartE2EDuration="29.170568336s" totalImagesPullingTime="21.231789274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:32.169715915 +0000 UTC m=+29.702318215" watchObservedRunningTime="2026-07-10 13:10:32.170568336 +0000 UTC m=+29.703170657" Jul 10 13:10:32.967476 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:32.967451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:32.968006 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:32.967566 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:33.275286 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.275257 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-28kx4"] Jul 10 13:10:33.275424 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.275362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:33.275475 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:33.275447 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:33.278750 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.278727 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nmbs2"] Jul 10 13:10:33.278857 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.278828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:33.278943 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:33.278923 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:33.279354 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.279335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t95jr"] Jul 10 13:10:33.279434 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:33.279418 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:33.279531 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:33.279510 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:34.126850 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:34.126814 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="39d0a8fc3121a0798340f203c65d4126e7f4d90d8d820e780090523f10923bae" exitCode=0 Jul 10 13:10:34.127260 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:34.126848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"39d0a8fc3121a0798340f203c65d4126e7f4d90d8d820e780090523f10923bae"} Jul 10 13:10:34.963178 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:34.963149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:34.963178 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:34.963165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:34.963407 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:34.963165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:34.963407 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:34.963239 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:34.963407 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:34.963336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:34.963407 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:34.963404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:36.133501 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.133416 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="cdaaa6a8292ac55fc40037a23cebf5d7c3136b5a91a0e81d2d9f581708c0b97d" exitCode=0 Jul 10 13:10:36.133501 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.133481 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"cdaaa6a8292ac55fc40037a23cebf5d7c3136b5a91a0e81d2d9f581708c0b97d"} Jul 10 13:10:36.618683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.618591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:36.618683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.618648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:36.618856 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.618724 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:36.618856 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.618726 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:36.618856 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.618773 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret podName:36ab361d-d9c8-4d97-bf03-4393b79c32f8 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:08.618760613 +0000 UTC m=+66.151362890 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret") pod "global-pull-secret-syncer-28kx4" (UID: "36ab361d-d9c8-4d97-bf03-4393b79c32f8") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:36.618856 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.618787 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs podName:7bbcdad2-6da8-4797-b9c0-f07a6964baf9 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:08.618780658 +0000 UTC m=+66.151382934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs") pod "network-metrics-daemon-t95jr" (UID: "7bbcdad2-6da8-4797-b9c0-f07a6964baf9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:36.719783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.719753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:36.719963 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.719879 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:36.719963 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.719894 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:36.719963 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.719903 2573 projected.go:196] Error preparing data for projected volume kube-api-access-t8mb6 for pod openshift-network-diagnostics/network-check-target-nmbs2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:36.719963 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.719951 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6 podName:75bfff61-c6ee-4ccc-96a8-a314e2b35abf nodeName:}" failed. No retries permitted until 2026-07-10 13:11:08.719939461 +0000 UTC m=+66.252541738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8mb6" (UniqueName: "kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6") pod "network-check-target-nmbs2" (UID: "75bfff61-c6ee-4ccc-96a8-a314e2b35abf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:36.963511 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.963432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:36.963511 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.963458 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:36.963511 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:36.963432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:36.963809 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.963525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:36.963809 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.963608 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:36.963809 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:36.963683 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:38.963677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:38.963422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:38.964162 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:38.963431 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:38.964162 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:38.963463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:38.964162 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:38.963785 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-28kx4" podUID="36ab361d-d9c8-4d97-bf03-4393b79c32f8" Jul 10 13:10:38.964162 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:38.963935 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t95jr" podUID="7bbcdad2-6da8-4797-b9c0-f07a6964baf9" Jul 10 13:10:38.964162 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:38.964005 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmbs2" podUID="75bfff61-c6ee-4ccc-96a8-a314e2b35abf" Jul 10 13:10:40.280779 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.280700 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-249.ec2.internal" event="NodeReady" Jul 10 13:10:40.281202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.280830 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 13:10:40.314445 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.314413 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:10:40.316466 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.316444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.318126 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.318057 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g"] Jul 10 13:10:40.319419 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.319387 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 13:10:40.319544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.319527 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 13:10:40.319620 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.319603 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ngc6m\"" Jul 10 13:10:40.319796 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.319776 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 13:10:40.320549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.319989 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg"] Jul 10 13:10:40.320549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.320070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.321585 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.321560 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk"] Jul 10 13:10:40.321819 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.321798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" Jul 10 13:10:40.323676 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.323658 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5"] Jul 10 13:10:40.323880 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.323850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.324988 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.324785 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hkn8r\"" Jul 10 13:10:40.324988 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.324858 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.325133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.325088 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 13:10:40.325351 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.325297 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.325850 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.325778 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-jdskh"] Jul 10 13:10:40.325986 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.325968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.326436 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.326421 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 13:10:40.326523 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.326484 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.326583 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.326421 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.328218 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.328197 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-75574\"" Jul 10 13:10:40.328310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.328241 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lr8vh\"" Jul 10 13:10:40.328310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.328254 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-755b7d5488-4vbsm"] Jul 10 13:10:40.328408 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.328322 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.330075 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.330000 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.330174 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.330141 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.330473 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.330456 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.331188 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.331167 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wvglq\"" Jul 10 13:10:40.331652 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.331617 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 13:10:40.331762 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.331745 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332253 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332515 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332552 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.333110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.332924 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.334269 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.334245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 13:10:40.335751 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.335729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pv9tz\"" Jul 10 13:10:40.335907 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.335893 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 13:10:40.339896 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.339724 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 13:10:40.340074 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.340035 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 13:10:40.340278 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.340255 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qk7gx"] Jul 10 13:10:40.340512 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.340490 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 13:10:40.340612 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.340526 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h8rcs\"" Jul 10 13:10:40.341396 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.341377 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 13:10:40.341688 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.341671 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 13:10:40.341795 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.341781 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.341919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.341880 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.342687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.342668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.343683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.342817 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf"] Jul 10 13:10:40.344793 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.344773 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-24fqk"] Jul 10 13:10:40.344903 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.344885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.346560 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.346545 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 13:10:40.346695 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.346678 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hhpjb"] Jul 10 13:10:40.346803 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.346786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.346892 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.346875 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 13:10:40.346892 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.346886 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7p5lh\"" Jul 10 13:10:40.348761 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.348735 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h"] Jul 10 13:10:40.348861 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.348774 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.348861 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.348795 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 13:10:40.348861 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.348802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.348861 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.348808 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.349487 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.349335 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 13:10:40.349487 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.349400 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 13:10:40.349487 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.349459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.349675 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.349622 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.349824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.349790 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 13:10:40.350052 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.350035 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zbtbv\"" Jul 10 13:10:40.351116 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.351090 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjh5l"] Jul 10 13:10:40.351366 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.351349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" Jul 10 13:10:40.352039 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.351937 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.352039 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.351960 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 13:10:40.352184 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.352082 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ndhrp\"" Jul 10 13:10:40.352184 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.352148 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm96w\"" Jul 10 13:10:40.352423 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.352404 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.353289 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.353260 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg"] Jul 10 13:10:40.353390 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.353347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.355294 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.355255 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 13:10:40.355719 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.355558 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 13:10:40.355719 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.355683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 13:10:40.356681 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.356664 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pnbzg\"" Jul 10 13:10:40.357205 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.356731 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 13:10:40.357205 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.356778 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vrvn5\"" Jul 10 13:10:40.359588 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.359461 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 13:10:40.360791 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.360767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5"] Jul 10 13:10:40.370656 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.369044 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:10:40.370656 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.369133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hhpjb"] Jul 10 13:10:40.372645 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.372612 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-755b7d5488-4vbsm"] Jul 10 13:10:40.372805 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.372790 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk"] Jul 10 13:10:40.373045 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.373023 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g"] Jul 10 13:10:40.374786 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.374770 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-24fqk"] Jul 10 13:10:40.376355 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.376115 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf"] Jul 10 13:10:40.377223 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.376925 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qk7gx"] Jul 10 13:10:40.378614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.378592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h"] Jul 10 13:10:40.383067 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.383051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jdskh"] Jul 10 13:10:40.383958 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.383941 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjh5l"] Jul 10 13:10:40.448990 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.448968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.449109 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q2m\" (UniqueName: \"kubernetes.io/projected/d6a98def-23d5-404f-8429-18347a24e8b9-kube-api-access-l4q2m\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.449109 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e571d6b8-14f6-4e5d-906b-819d485b0117-cert\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.449206 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.449206 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-trusted-ca\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.449291 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2beeec8-8f39-4756-8e53-4403d5c4d19c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.449291 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.449291 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e79848-1a24-43f0-bee6-1590702cfaca-serving-cert\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.449403 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc44162-1c8b-427f-a7fb-ac64c02eb3af-tmp-dir\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.449403 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/412665cf-e9c5-4da8-897f-5b5d82876e51-config\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.449403 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgz28\" (UniqueName: \"kubernetes.io/projected/b4e79848-1a24-43f0-bee6-1590702cfaca-kube-api-access-wgz28\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.449403 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.449553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2r2t\" (UniqueName: \"kubernetes.io/projected/e2beeec8-8f39-4756-8e53-4403d5c4d19c-kube-api-access-r2r2t\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.449553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkrg\" (UniqueName: \"kubernetes.io/projected/412665cf-e9c5-4da8-897f-5b5d82876e51-kube-api-access-qvkrg\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.449553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2beeec8-8f39-4756-8e53-4403d5c4d19c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.449686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6wt\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.449686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cltn\" (UniqueName: \"kubernetes.io/projected/b1c504e0-9a9e-4886-b3eb-a582f363b55e-kube-api-access-7cltn\") pod \"network-check-source-869dcfcdcd-7pj2h\" (UID: \"b1c504e0-9a9e-4886-b3eb-a582f363b55e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" Jul 10 13:10:40.449686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.449686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-default-certificate\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.449686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-stats-auth\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b364a518-6d8e-4b67-b242-d64942c29b50-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-config\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-snapshots\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dzf\" (UniqueName: \"kubernetes.io/projected/ca3745bd-546f-4cc6-8edf-672d5511f623-kube-api-access-97dzf\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.449905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtps9\" (UniqueName: \"kubernetes.io/projected/a41dee19-9297-424c-a35e-41dbf2f882c7-kube-api-access-qtps9\") pod \"volume-data-source-validator-78495597f6-vvthg\" (UID: \"a41dee19-9297-424c-a35e-41dbf2f882c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.449984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqs4\" (UniqueName: \"kubernetes.io/projected/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-kube-api-access-4qqs4\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-tmp\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6a98def-23d5-404f-8429-18347a24e8b9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afc44162-1c8b-427f-a7fb-ac64c02eb3af-config-volume\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.450202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jt7\" (UniqueName: \"kubernetes.io/projected/afc44162-1c8b-427f-a7fb-ac64c02eb3af-kube-api-access-d5jt7\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/412665cf-e9c5-4da8-897f-5b5d82876e51-serving-cert\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxt9j\" (UniqueName: \"kubernetes.io/projected/e571d6b8-14f6-4e5d-906b-819d485b0117-kube-api-access-zxt9j\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcs4\" (UniqueName: \"kubernetes.io/projected/509c9cde-d447-4dd4-b153-37eb06725a6a-kube-api-access-vkcs4\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.450542 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.450517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509c9cde-d447-4dd4-b153-37eb06725a6a-serving-cert\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.551561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkrg\" (UniqueName: \"kubernetes.io/projected/412665cf-e9c5-4da8-897f-5b5d82876e51-kube-api-access-qvkrg\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.551561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2beeec8-8f39-4756-8e53-4403d5c4d19c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.551561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6wt\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cltn\" (UniqueName: \"kubernetes.io/projected/b1c504e0-9a9e-4886-b3eb-a582f363b55e-kube-api-access-7cltn\") pod \"network-check-source-869dcfcdcd-7pj2h\" (UID: \"b1c504e0-9a9e-4886-b3eb-a582f363b55e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-default-certificate\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-stats-auth\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.551767 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:40.551873 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.551831 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.051811985 +0000 UTC m=+38.584414282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b364a518-6d8e-4b67-b242-d64942c29b50-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-config\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.551985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-snapshots\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dzf\" (UniqueName: \"kubernetes.io/projected/ca3745bd-546f-4cc6-8edf-672d5511f623-kube-api-access-97dzf\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtps9\" (UniqueName: \"kubernetes.io/projected/a41dee19-9297-424c-a35e-41dbf2f882c7-kube-api-access-qtps9\") pod \"volume-data-source-validator-78495597f6-vvthg\" (UID: \"a41dee19-9297-424c-a35e-41dbf2f882c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.552224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqs4\" (UniqueName: \"kubernetes.io/projected/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-kube-api-access-4qqs4\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-tmp\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6a98def-23d5-404f-8429-18347a24e8b9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afc44162-1c8b-427f-a7fb-ac64c02eb3af-config-volume\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jt7\" (UniqueName: \"kubernetes.io/projected/afc44162-1c8b-427f-a7fb-ac64c02eb3af-kube-api-access-d5jt7\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/412665cf-e9c5-4da8-897f-5b5d82876e51-serving-cert\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxt9j\" (UniqueName: \"kubernetes.io/projected/e571d6b8-14f6-4e5d-906b-819d485b0117-kube-api-access-zxt9j\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552566 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b364a518-6d8e-4b67-b242-d64942c29b50-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcs4\" (UniqueName: \"kubernetes.io/projected/509c9cde-d447-4dd4-b153-37eb06725a6a-kube-api-access-vkcs4\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.552663 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509c9cde-d447-4dd4-b153-37eb06725a6a-serving-cert\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.552748 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.552725 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.052707555 +0000 UTC m=+38.585309844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q2m\" (UniqueName: \"kubernetes.io/projected/d6a98def-23d5-404f-8429-18347a24e8b9-kube-api-access-l4q2m\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552775 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-config\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.552835 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.552859 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.052852136 +0000 UTC m=+38.585454413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e571d6b8-14f6-4e5d-906b-819d485b0117-cert\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.552976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-trusted-ca\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2beeec8-8f39-4756-8e53-4403d5c4d19c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.553087 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.05306981 +0000 UTC m=+38.585672118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e79848-1a24-43f0-bee6-1590702cfaca-serving-cert\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.553548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc44162-1c8b-427f-a7fb-ac64c02eb3af-tmp-dir\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/412665cf-e9c5-4da8-897f-5b5d82876e51-config\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz28\" (UniqueName: \"kubernetes.io/projected/b4e79848-1a24-43f0-bee6-1590702cfaca-kube-api-access-wgz28\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r2t\" (UniqueName: \"kubernetes.io/projected/e2beeec8-8f39-4756-8e53-4403d5c4d19c-kube-api-access-r2r2t\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6a98def-23d5-404f-8429-18347a24e8b9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553773 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-tmp\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.554319 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.553820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/509c9cde-d447-4dd4-b153-37eb06725a6a-trusted-ca\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.554718 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.554322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2beeec8-8f39-4756-8e53-4403d5c4d19c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.554718 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.554392 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:40.554718 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.554410 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:40.554718 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.554459 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.054443437 +0000 UTC m=+38.587045732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:40.554944 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.554924 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b4e79848-1a24-43f0-bee6-1590702cfaca-snapshots\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.555108 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.555081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afc44162-1c8b-427f-a7fb-ac64c02eb3af-config-volume\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.555349 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.555326 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e79848-1a24-43f0-bee6-1590702cfaca-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.555446 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.555430 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:40.555507 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.555492 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.055475595 +0000 UTC m=+38.588077891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:40.555692 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.555669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/412665cf-e9c5-4da8-897f-5b5d82876e51-config\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.555692 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.555683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.555955 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.555921 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afc44162-1c8b-427f-a7fb-ac64c02eb3af-tmp-dir\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.556164 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.556135 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:40.561643 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.561607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.562051 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.561961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509c9cde-d447-4dd4-b153-37eb06725a6a-serving-cert\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.562231 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.562191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/412665cf-e9c5-4da8-897f-5b5d82876e51-serving-cert\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.562437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.562419 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2beeec8-8f39-4756-8e53-4403d5c4d19c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.562553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.562527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.562893 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.562873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-stats-auth\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.563038 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.562999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.563111 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:40.563061 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:41.059043433 +0000 UTC m=+38.591645725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:40.563213 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.563194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e79848-1a24-43f0-bee6-1590702cfaca-serving-cert\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.563571 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.563501 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-default-certificate\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.565818 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.556215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.566352 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.566317 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cltn\" (UniqueName: \"kubernetes.io/projected/b1c504e0-9a9e-4886-b3eb-a582f363b55e-kube-api-access-7cltn\") pod \"network-check-source-869dcfcdcd-7pj2h\" (UID: \"b1c504e0-9a9e-4886-b3eb-a582f363b55e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" Jul 10 13:10:40.566492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.566472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e571d6b8-14f6-4e5d-906b-819d485b0117-cert\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.570056 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.569741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6wt\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.572074 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.571242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jt7\" (UniqueName: \"kubernetes.io/projected/afc44162-1c8b-427f-a7fb-ac64c02eb3af-kube-api-access-d5jt7\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:40.573437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.573413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2r2t\" (UniqueName: \"kubernetes.io/projected/e2beeec8-8f39-4756-8e53-4403d5c4d19c-kube-api-access-r2r2t\") pod \"kube-storage-version-migrator-operator-799cb7f958-zmnmf\" (UID: \"e2beeec8-8f39-4756-8e53-4403d5c4d19c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.574201 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.574169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:40.574355 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.574333 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqs4\" (UniqueName: \"kubernetes.io/projected/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-kube-api-access-4qqs4\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:40.574890 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.574858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz28\" (UniqueName: \"kubernetes.io/projected/b4e79848-1a24-43f0-bee6-1590702cfaca-kube-api-access-wgz28\") pod \"insights-operator-68d9dd94cf-24fqk\" (UID: \"b4e79848-1a24-43f0-bee6-1590702cfaca\") " pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.574980 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.574951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcs4\" (UniqueName: \"kubernetes.io/projected/509c9cde-d447-4dd4-b153-37eb06725a6a-kube-api-access-vkcs4\") pod \"console-operator-7866497494-jdskh\" (UID: \"509c9cde-d447-4dd4-b153-37eb06725a6a\") " pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.575022 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.574991 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q2m\" (UniqueName: \"kubernetes.io/projected/d6a98def-23d5-404f-8429-18347a24e8b9-kube-api-access-l4q2m\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:40.575069 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.575047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dzf\" (UniqueName: \"kubernetes.io/projected/ca3745bd-546f-4cc6-8edf-672d5511f623-kube-api-access-97dzf\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:40.575580 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.575561 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxt9j\" (UniqueName: \"kubernetes.io/projected/e571d6b8-14f6-4e5d-906b-819d485b0117-kube-api-access-zxt9j\") pod \"ingress-canary-hhpjb\" (UID: \"e571d6b8-14f6-4e5d-906b-819d485b0117\") " pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.576072 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.576051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtps9\" (UniqueName: \"kubernetes.io/projected/a41dee19-9297-424c-a35e-41dbf2f882c7-kube-api-access-qtps9\") pod \"volume-data-source-validator-78495597f6-vvthg\" (UID: \"a41dee19-9297-424c-a35e-41dbf2f882c7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" Jul 10 13:10:40.576495 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.576455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkrg\" (UniqueName: \"kubernetes.io/projected/412665cf-e9c5-4da8-897f-5b5d82876e51-kube-api-access-qvkrg\") pod \"service-ca-operator-7695fc6c8c-hrq4g\" (UID: \"412665cf-e9c5-4da8-897f-5b5d82876e51\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.641360 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.641336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" Jul 10 13:10:40.648715 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.648692 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" Jul 10 13:10:40.678909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.678874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:40.700245 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.700218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" Jul 10 13:10:40.708993 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.708972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" Jul 10 13:10:40.715575 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.715555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hhpjb" Jul 10 13:10:40.722212 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.722189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" Jul 10 13:10:40.962659 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.962570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:10:40.962659 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.962575 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:10:40.962931 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.962906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:10:40.965401 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.965378 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnrp9\"" Jul 10 13:10:40.965491 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.965457 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xcvz\"" Jul 10 13:10:40.965491 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.965482 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:10:40.965614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:40.965589 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:10:41.058772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.058739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:41.058918 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.058783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:41.058956 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.058914 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:41.059018 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059005 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.058982509 +0000 UTC m=+39.591584798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:41.059018 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059004 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:41.059114 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.058918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:41.059114 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059037 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.059020309 +0000 UTC m=+39.591622609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:41.059114 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059074 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.059063505 +0000 UTC m=+39.591665785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:41.059114 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.059098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:41.059246 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.059152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:41.059246 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.059199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:41.059246 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059216 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:41.059245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059262 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.059249166 +0000 UTC m=+39.591851444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059300 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059319 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059333 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:41.059369 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059304 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:41.059605 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059356 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.059341435 +0000 UTC m=+39.591943713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:41.059605 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059401 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.05938421 +0000 UTC m=+39.591986503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:41.059605 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:41.059422 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:42.059410623 +0000 UTC m=+39.592012906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.073506 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.073583 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.073561507 +0000 UTC m=+41.606163789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.073792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.073932 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.073978 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.073963868 +0000 UTC m=+41.606566147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074374 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074388 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074402 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074430 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.07441596 +0000 UTC m=+41.607018239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:42.074729 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074463 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.074447055 +0000 UTC m=+41.607049333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:42.076248 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074492 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:42.076248 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074521 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.074512553 +0000 UTC m=+41.607114829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:42.076248 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074537 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.074525996 +0000 UTC m=+41.607128273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:42.076248 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074579 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:42.076248 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:42.074608 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:44.074598835 +0000 UTC m=+41.607201113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:42.139228 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.138792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h"] Jul 10 13:10:42.148787 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.148756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jdskh"] Jul 10 13:10:42.155174 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.155148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerStarted","Data":"d64a312ddd9dc960f90c85168835133eb0660d9d1c10dee7adb169176a21507e"} Jul 10 13:10:42.164237 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.164207 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg"] Jul 10 13:10:42.173886 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.173847 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf"] Jul 10 13:10:42.175092 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.175071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hhpjb"] Jul 10 13:10:42.176241 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.176221 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-24fqk"] Jul 10 13:10:42.187734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:42.187708 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g"] Jul 10 13:10:42.190747 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.190721 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509c9cde_d447_4dd4_b153_37eb06725a6a.slice/crio-9ebc027056711e85fb5107149b6ae7a29ff39f89e929e531fa2ef5baf0a5caa1 WatchSource:0}: Error finding container 9ebc027056711e85fb5107149b6ae7a29ff39f89e929e531fa2ef5baf0a5caa1: Status 404 returned error can't find the container with id 9ebc027056711e85fb5107149b6ae7a29ff39f89e929e531fa2ef5baf0a5caa1 Jul 10 13:10:42.191247 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.191211 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2beeec8_8f39_4756_8e53_4403d5c4d19c.slice/crio-cef352838d2b06113d34d8010930108aad83bc0dce473f6ac2a46b4966b7eccc WatchSource:0}: Error finding container cef352838d2b06113d34d8010930108aad83bc0dce473f6ac2a46b4966b7eccc: Status 404 returned error can't find the container with id cef352838d2b06113d34d8010930108aad83bc0dce473f6ac2a46b4966b7eccc Jul 10 13:10:42.192083 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.192064 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e79848_1a24_43f0_bee6_1590702cfaca.slice/crio-96ef081a2a5d694857053c2bad6cf599544785d22d1b29faaebd2df909626422 WatchSource:0}: Error finding container 96ef081a2a5d694857053c2bad6cf599544785d22d1b29faaebd2df909626422: Status 404 returned error can't find the container with id 96ef081a2a5d694857053c2bad6cf599544785d22d1b29faaebd2df909626422 Jul 10 13:10:42.194821 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.192939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41dee19_9297_424c_a35e_41dbf2f882c7.slice/crio-8a3fb6582d4b2742e6fda8bbc682bdcd311a6197b273eb7671e6586199ee3588 WatchSource:0}: Error finding container 8a3fb6582d4b2742e6fda8bbc682bdcd311a6197b273eb7671e6586199ee3588: Status 404 returned error can't find the container with id 8a3fb6582d4b2742e6fda8bbc682bdcd311a6197b273eb7671e6586199ee3588 Jul 10 13:10:42.194821 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.194257 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode571d6b8_14f6_4e5d_906b_819d485b0117.slice/crio-9c7f32101e39875e5323645a2812ff0574fc0c547e578c1e281032a38849b408 WatchSource:0}: Error finding container 9c7f32101e39875e5323645a2812ff0574fc0c547e578c1e281032a38849b408: Status 404 returned error can't find the container with id 9c7f32101e39875e5323645a2812ff0574fc0c547e578c1e281032a38849b408 Jul 10 13:10:42.195305 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.194962 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c504e0_9a9e_4886_b3eb_a582f363b55e.slice/crio-a21e88b6a27e3d134b8108ef87dba4ab87388c0bbd2dc27bf4bcbee464326d11 WatchSource:0}: Error finding container a21e88b6a27e3d134b8108ef87dba4ab87388c0bbd2dc27bf4bcbee464326d11: Status 404 returned error can't find the container with id a21e88b6a27e3d134b8108ef87dba4ab87388c0bbd2dc27bf4bcbee464326d11 Jul 10 13:10:42.196028 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:42.195961 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412665cf_e9c5_4da8_897f_5b5d82876e51.slice/crio-b52142938039065983cd6e777c4bc90ea8d8b5c95c9c198d4ef3bc519040afc2 WatchSource:0}: Error finding container b52142938039065983cd6e777c4bc90ea8d8b5c95c9c198d4ef3bc519040afc2: Status 404 returned error can't find the container with id b52142938039065983cd6e777c4bc90ea8d8b5c95c9c198d4ef3bc519040afc2 Jul 10 13:10:43.164112 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.164028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" event={"ID":"b4e79848-1a24-43f0-bee6-1590702cfaca","Type":"ContainerStarted","Data":"96ef081a2a5d694857053c2bad6cf599544785d22d1b29faaebd2df909626422"} Jul 10 13:10:43.166452 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.166335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hhpjb" event={"ID":"e571d6b8-14f6-4e5d-906b-819d485b0117","Type":"ContainerStarted","Data":"9c7f32101e39875e5323645a2812ff0574fc0c547e578c1e281032a38849b408"} Jul 10 13:10:43.168564 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.168533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" event={"ID":"412665cf-e9c5-4da8-897f-5b5d82876e51","Type":"ContainerStarted","Data":"b52142938039065983cd6e777c4bc90ea8d8b5c95c9c198d4ef3bc519040afc2"} Jul 10 13:10:43.170264 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.170190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" event={"ID":"b1c504e0-9a9e-4886-b3eb-a582f363b55e","Type":"ContainerStarted","Data":"a21e88b6a27e3d134b8108ef87dba4ab87388c0bbd2dc27bf4bcbee464326d11"} Jul 10 13:10:43.173078 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.173005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" event={"ID":"e2beeec8-8f39-4756-8e53-4403d5c4d19c","Type":"ContainerStarted","Data":"cef352838d2b06113d34d8010930108aad83bc0dce473f6ac2a46b4966b7eccc"} Jul 10 13:10:43.174827 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.174761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" event={"ID":"a41dee19-9297-424c-a35e-41dbf2f882c7","Type":"ContainerStarted","Data":"8a3fb6582d4b2742e6fda8bbc682bdcd311a6197b273eb7671e6586199ee3588"} Jul 10 13:10:43.176465 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.176367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jdskh" event={"ID":"509c9cde-d447-4dd4-b153-37eb06725a6a","Type":"ContainerStarted","Data":"9ebc027056711e85fb5107149b6ae7a29ff39f89e929e531fa2ef5baf0a5caa1"} Jul 10 13:10:43.185517 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.184619 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="d64a312ddd9dc960f90c85168835133eb0660d9d1c10dee7adb169176a21507e" exitCode=0 Jul 10 13:10:43.185517 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:43.184670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"d64a312ddd9dc960f90c85168835133eb0660d9d1c10dee7adb169176a21507e"} Jul 10 13:10:44.093014 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.092973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:44.093211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:44.093211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:44.093211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:44.093211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:44.093428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:44.093428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.093286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:44.093520 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.093450 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.093430543 +0000 UTC m=+45.626032824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:44.093908 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.093888 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:44.094003 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.093948 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.093930982 +0000 UTC m=+45.626533261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:44.094064 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094011 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:44.094064 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094045 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.09403482 +0000 UTC m=+45.626637103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:44.094165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094106 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:44.094165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094118 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:44.094165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094149 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.094138621 +0000 UTC m=+45.626740901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:44.094349 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094216 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:44.094349 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094248 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.094237849 +0000 UTC m=+45.626840129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:44.094349 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094296 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:44.094349 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094324 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.094314687 +0000 UTC m=+45.626916967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:44.094545 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094369 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:44.094545 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:44.094398 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:48.094388839 +0000 UTC m=+45.626991119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:44.197775 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.197742 2573 generic.go:356] "Generic (PLEG): container finished" podID="82bbaed0-7855-4587-b7c4-2d4b15ce23ea" containerID="fd10ba3d263d70849bdd7842b9e0af7c2b8111bbfe06a4720f372f64a26e50bd" exitCode=0 Jul 10 13:10:44.198201 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:44.197830 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerDied","Data":"fd10ba3d263d70849bdd7842b9e0af7c2b8111bbfe06a4720f372f64a26e50bd"} Jul 10 13:10:45.202200 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:45.201994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" event={"ID":"a41dee19-9297-424c-a35e-41dbf2f882c7","Type":"ContainerStarted","Data":"8cbd9f1badf440016c63ee5197184ef40303038b7b3375dbebbdb90b3f530566"} Jul 10 13:10:45.205711 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:45.205678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" event={"ID":"82bbaed0-7855-4587-b7c4-2d4b15ce23ea","Type":"ContainerStarted","Data":"92a1120c41836af85c2aa6d429b713d4e15c63aa757c706b7eb15e6d80f10022"} Jul 10 13:10:45.220195 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:45.220148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vvthg" podStartSLOduration=34.889724711 podStartE2EDuration="37.220131391s" totalImagesPullingTime="2.33040668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:45.220011266 +0000 UTC m=+42.752613567" watchObservedRunningTime="2026-07-10 13:10:45.220131391 +0000 UTC m=+42.752733692" Jul 10 13:10:45.245018 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:45.244962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5qsqd" podStartSLOduration=10.357139285 podStartE2EDuration="42.244948078s" totalImagesPullingTime="31.887808793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:45.243900645 +0000 UTC m=+42.776502945" watchObservedRunningTime="2026-07-10 13:10:45.244948078 +0000 UTC m=+42.777550376" Jul 10 13:10:48.131921 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.131881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.131944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.131995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132040 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132065 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132108 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.132063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132139 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132157 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132141 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.132120397 +0000 UTC m=+53.664722690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132243 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.132219933 +0000 UTC m=+53.664822214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132262 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.13225296 +0000 UTC m=+53.664855237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132296 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.132287722 +0000 UTC m=+53.664890001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.132351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.132389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:48.132684 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:48.132415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:48.133230 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132458 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.13244718 +0000 UTC m=+53.665049478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:48.133230 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132486 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:48.133230 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132528 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.132514631 +0000 UTC m=+53.665116917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:48.133230 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132536 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:48.133230 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:48.132574 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:10:56.1325631 +0000 UTC m=+53.665165397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:50.223685 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.223385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hhpjb" event={"ID":"e571d6b8-14f6-4e5d-906b-819d485b0117","Type":"ContainerStarted","Data":"e2368622910a98d5b68cb4fafa8b01cc9d12709dc99d5fc2735f32225d5a0fcc"} Jul 10 13:10:50.225462 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.225433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" event={"ID":"412665cf-e9c5-4da8-897f-5b5d82876e51","Type":"ContainerStarted","Data":"bebd57979551325ca578e1d7a7e21b369274ab5d0993add2d8d094f03e134aa0"} Jul 10 13:10:50.228543 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.228131 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" event={"ID":"b1c504e0-9a9e-4886-b3eb-a582f363b55e","Type":"ContainerStarted","Data":"b59a28b99796811abe3814c7b11042cd32a4d56b6635a23266796b74c2f1af7f"} Jul 10 13:10:50.230263 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.229858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" event={"ID":"e2beeec8-8f39-4756-8e53-4403d5c4d19c","Type":"ContainerStarted","Data":"468d73638ffcab94d7ab8edd34af36084eff93b742ee24b3593ed4fb8ba73e00"} Jul 10 13:10:50.232691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.231880 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/0.log" Jul 10 13:10:50.232691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.231933 2573 generic.go:356] "Generic (PLEG): container finished" podID="509c9cde-d447-4dd4-b153-37eb06725a6a" containerID="bc9099a07d4dfa59c4ca0a37f7b47de7bec2156fbe7f2370eab77fb6bcaf5ca5" exitCode=255 Jul 10 13:10:50.232691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.232009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jdskh" event={"ID":"509c9cde-d447-4dd4-b153-37eb06725a6a","Type":"ContainerDied","Data":"bc9099a07d4dfa59c4ca0a37f7b47de7bec2156fbe7f2370eab77fb6bcaf5ca5"} Jul 10 13:10:50.232691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.232565 2573 scope.go:117] "RemoveContainer" containerID="bc9099a07d4dfa59c4ca0a37f7b47de7bec2156fbe7f2370eab77fb6bcaf5ca5" Jul 10 13:10:50.234208 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.234186 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" event={"ID":"b4e79848-1a24-43f0-bee6-1590702cfaca","Type":"ContainerStarted","Data":"8fe7ee86ede0c2d4e1013977ca4ab879ab15da124a7347985cddc16ea6198f17"} Jul 10 13:10:50.240983 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.240655 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hhpjb" podStartSLOduration=2.8327793850000003 podStartE2EDuration="10.240623463s" totalImagesPullingTime="7.407844078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:50.239918437 +0000 UTC m=+47.772520735" watchObservedRunningTime="2026-07-10 13:10:50.240623463 +0000 UTC m=+47.773225764" Jul 10 13:10:50.287673 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.286707 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" podStartSLOduration=34.879909718 podStartE2EDuration="42.28668838s" totalImagesPullingTime="7.406778662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:50.268313944 +0000 UTC m=+47.800916254" watchObservedRunningTime="2026-07-10 13:10:50.28668838 +0000 UTC m=+47.819290679" Jul 10 13:10:50.309049 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.309005 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" podStartSLOduration=34.898040915 podStartE2EDuration="42.308985792s" totalImagesPullingTime="7.410944877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:50.308767334 +0000 UTC m=+47.841369634" watchObservedRunningTime="2026-07-10 13:10:50.308985792 +0000 UTC m=+47.841588091" Jul 10 13:10:50.324085 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.324039 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-7pj2h" podStartSLOduration=34.906771927 podStartE2EDuration="42.324014906s" totalImagesPullingTime="7.417242979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:50.322604636 +0000 UTC m=+47.855206938" watchObservedRunningTime="2026-07-10 13:10:50.324014906 +0000 UTC m=+47.856617211" Jul 10 13:10:50.340000 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.339943 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" podStartSLOduration=34.929507985 podStartE2EDuration="42.339924084s" totalImagesPullingTime="7.410416099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:50.337877107 +0000 UTC m=+47.870479405" watchObservedRunningTime="2026-07-10 13:10:50.339924084 +0000 UTC m=+47.872526384" Jul 10 13:10:50.679279 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.679249 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:50.679414 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:50.679345 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:10:51.239950 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.239924 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:10:51.240404 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.240345 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/0.log" Jul 10 13:10:51.240404 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.240375 2573 generic.go:356] "Generic (PLEG): container finished" podID="509c9cde-d447-4dd4-b153-37eb06725a6a" containerID="b0141ad0576aa5f54dedab921d19392ece748453e46105c79275ac6644930724" exitCode=255 Jul 10 13:10:51.240506 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.240453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jdskh" event={"ID":"509c9cde-d447-4dd4-b153-37eb06725a6a","Type":"ContainerDied","Data":"b0141ad0576aa5f54dedab921d19392ece748453e46105c79275ac6644930724"} Jul 10 13:10:51.240562 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.240504 2573 scope.go:117] "RemoveContainer" containerID="bc9099a07d4dfa59c4ca0a37f7b47de7bec2156fbe7f2370eab77fb6bcaf5ca5" Jul 10 13:10:51.240753 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.240734 2573 scope.go:117] "RemoveContainer" containerID="b0141ad0576aa5f54dedab921d19392ece748453e46105c79275ac6644930724" Jul 10 13:10:51.240943 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:51.240916 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jdskh_openshift-console-operator(509c9cde-d447-4dd4-b153-37eb06725a6a)\"" pod="openshift-console-operator/console-operator-7866497494-jdskh" podUID="509c9cde-d447-4dd4-b153-37eb06725a6a" Jul 10 13:10:51.377669 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.377618 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vmhkk"] Jul 10 13:10:51.408301 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.408276 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmhkk"] Jul 10 13:10:51.408464 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.408376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.410708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.410684 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 13:10:51.410849 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.410733 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 13:10:51.411013 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.410996 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4zgls\"" Jul 10 13:10:51.466924 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.466896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.467048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.466958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qh97\" (UniqueName: \"kubernetes.io/projected/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-api-access-8qh97\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.467140 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.467120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44f5d802-1f73-4843-ac3e-c7985f017eb6-data-volume\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.467195 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.467158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.467242 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.467215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44f5d802-1f73-4843-ac3e-c7985f017eb6-crio-socket\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.567972 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.567913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568076 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.567967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qh97\" (UniqueName: \"kubernetes.io/projected/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-api-access-8qh97\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568138 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:51.568075 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:10:51.568191 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:51.568143 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:52.068124259 +0000 UTC m=+49.600726548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : secret "insights-runtime-extractor-tls" not found Jul 10 13:10:51.568250 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.568197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44f5d802-1f73-4843-ac3e-c7985f017eb6-data-volume\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568250 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.568226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568334 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.568281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44f5d802-1f73-4843-ac3e-c7985f017eb6-crio-socket\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568391 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:51.568368 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:10:51.568437 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:51.568429 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:52.068410535 +0000 UTC m=+49.601012827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : configmap "kube-rbac-proxy" not found Jul 10 13:10:51.568489 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.568471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44f5d802-1f73-4843-ac3e-c7985f017eb6-data-volume\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.568554 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.568539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44f5d802-1f73-4843-ac3e-c7985f017eb6-crio-socket\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.579755 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.579737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qh97\" (UniqueName: \"kubernetes.io/projected/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-api-access-8qh97\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:51.840492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.840402 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz"] Jul 10 13:10:51.860569 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.860544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz"] Jul 10 13:10:51.860724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.860652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" Jul 10 13:10:51.862800 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.862773 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6n4np\"" Jul 10 13:10:51.862919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.862899 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:51.863002 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.862986 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 13:10:51.971514 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:51.971490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckzp8\" (UniqueName: \"kubernetes.io/projected/a2b4853b-41b6-4f9d-9136-17468cb36fa3-kube-api-access-ckzp8\") pod \"migrator-7f59b54f6c-gq7bz\" (UID: \"a2b4853b-41b6-4f9d-9136-17468cb36fa3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" Jul 10 13:10:52.072221 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.072193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:52.072365 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.072245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckzp8\" (UniqueName: \"kubernetes.io/projected/a2b4853b-41b6-4f9d-9136-17468cb36fa3-kube-api-access-ckzp8\") pod \"migrator-7f59b54f6c-gq7bz\" (UID: \"a2b4853b-41b6-4f9d-9136-17468cb36fa3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" Jul 10 13:10:52.072365 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.072279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:52.072365 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:52.072312 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:10:52.072471 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:52.072373 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:53.072355867 +0000 UTC m=+50.604958148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : configmap "kube-rbac-proxy" not found Jul 10 13:10:52.072471 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:52.072390 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:10:52.072471 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:52.072433 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:53.072422064 +0000 UTC m=+50.605024341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : secret "insights-runtime-extractor-tls" not found Jul 10 13:10:52.080247 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.080217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckzp8\" (UniqueName: \"kubernetes.io/projected/a2b4853b-41b6-4f9d-9136-17468cb36fa3-kube-api-access-ckzp8\") pod \"migrator-7f59b54f6c-gq7bz\" (UID: \"a2b4853b-41b6-4f9d-9136-17468cb36fa3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" Jul 10 13:10:52.170797 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.170769 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" Jul 10 13:10:52.247249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.247221 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:10:52.247670 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.247607 2573 scope.go:117] "RemoveContainer" containerID="b0141ad0576aa5f54dedab921d19392ece748453e46105c79275ac6644930724" Jul 10 13:10:52.247939 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:52.247914 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jdskh_openshift-console-operator(509c9cde-d447-4dd4-b153-37eb06725a6a)\"" pod="openshift-console-operator/console-operator-7866497494-jdskh" podUID="509c9cde-d447-4dd4-b153-37eb06725a6a" Jul 10 13:10:52.302430 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.302395 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz"] Jul 10 13:10:52.305713 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:52.305679 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b4853b_41b6_4f9d_9136_17468cb36fa3.slice/crio-64220e131fbd70ba742406c15a75d77770b8a473244d9ff1f54867c30a1211ea WatchSource:0}: Error finding container 64220e131fbd70ba742406c15a75d77770b8a473244d9ff1f54867c30a1211ea: Status 404 returned error can't find the container with id 64220e131fbd70ba742406c15a75d77770b8a473244d9ff1f54867c30a1211ea Jul 10 13:10:52.365776 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:52.365759 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zb9b_47874066-db3d-4c40-9c31-e4ee15f49de4/dns-node-resolver/0.log" Jul 10 13:10:53.081867 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.081841 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:53.082018 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.081955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:53.082018 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:53.081984 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:10:53.082092 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:53.082023 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:10:53.082092 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:53.082038 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:55.082023892 +0000 UTC m=+52.614626172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : secret "insights-runtime-extractor-tls" not found Jul 10 13:10:53.082092 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:53.082062 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:55.082050651 +0000 UTC m=+52.614652929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : configmap "kube-rbac-proxy" not found Jul 10 13:10:53.255175 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.255139 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" event={"ID":"a2b4853b-41b6-4f9d-9136-17468cb36fa3","Type":"ContainerStarted","Data":"64220e131fbd70ba742406c15a75d77770b8a473244d9ff1f54867c30a1211ea"} Jul 10 13:10:53.366419 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.366344 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bp48_c526be69-1738-4d64-8989-bb9c151b0ce6/node-ca/0.log" Jul 10 13:10:53.662829 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.662796 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-22qdg"] Jul 10 13:10:53.676032 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.676005 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-22qdg"] Jul 10 13:10:53.676182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.676116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.678473 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.678445 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 13:10:53.679173 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.679154 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rkxft\"" Jul 10 13:10:53.679303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.679228 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 13:10:53.679303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.679247 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 13:10:53.679303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.679232 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 13:10:53.789478 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.789448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4v9\" (UniqueName: \"kubernetes.io/projected/79709752-cad4-43ff-b5a8-3575462085a3-kube-api-access-zc4v9\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.789663 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.789509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79709752-cad4-43ff-b5a8-3575462085a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.789663 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.789626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79709752-cad4-43ff-b5a8-3575462085a3-signing-key\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.890986 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.890955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4v9\" (UniqueName: \"kubernetes.io/projected/79709752-cad4-43ff-b5a8-3575462085a3-kube-api-access-zc4v9\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.891136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.891017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79709752-cad4-43ff-b5a8-3575462085a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.891207 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.891143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79709752-cad4-43ff-b5a8-3575462085a3-signing-key\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.893770 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.893747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79709752-cad4-43ff-b5a8-3575462085a3-signing-key\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.899881 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.899838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4v9\" (UniqueName: \"kubernetes.io/projected/79709752-cad4-43ff-b5a8-3575462085a3-kube-api-access-zc4v9\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.904515 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.904421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79709752-cad4-43ff-b5a8-3575462085a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-22qdg\" (UID: \"79709752-cad4-43ff-b5a8-3575462085a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:53.985971 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:53.985910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" Jul 10 13:10:54.116492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:54.116466 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-22qdg"] Jul 10 13:10:54.168890 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:10:54.168856 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79709752_cad4_43ff_b5a8_3575462085a3.slice/crio-1395e969ec2691989ea20bb20c6f6a64526f16ab89a92f5ae1a898bf57a95431 WatchSource:0}: Error finding container 1395e969ec2691989ea20bb20c6f6a64526f16ab89a92f5ae1a898bf57a95431: Status 404 returned error can't find the container with id 1395e969ec2691989ea20bb20c6f6a64526f16ab89a92f5ae1a898bf57a95431 Jul 10 13:10:54.170058 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:54.170037 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hhpjb_e571d6b8-14f6-4e5d-906b-819d485b0117/serve-healthcheck-canary/0.log" Jul 10 13:10:54.262413 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:54.262387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" event={"ID":"79709752-cad4-43ff-b5a8-3575462085a3","Type":"ContainerStarted","Data":"1395e969ec2691989ea20bb20c6f6a64526f16ab89a92f5ae1a898bf57a95431"} Jul 10 13:10:55.101378 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.101341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:55.101572 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.101470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:55.101572 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:55.101520 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:10:55.101572 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:55.101555 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:10:55.101728 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:55.101604 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:59.101583273 +0000 UTC m=+56.634185553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : secret "insights-runtime-extractor-tls" not found Jul 10 13:10:55.101728 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:55.101626 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:59.101616149 +0000 UTC m=+56.634218440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : configmap "kube-rbac-proxy" not found Jul 10 13:10:55.270128 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.270092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" event={"ID":"a2b4853b-41b6-4f9d-9136-17468cb36fa3","Type":"ContainerStarted","Data":"c8aa6839457427d794f4c6fa16fd35a4a7dc3f1d86de82b6f4658ff7af391409"} Jul 10 13:10:55.270581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.270213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" event={"ID":"a2b4853b-41b6-4f9d-9136-17468cb36fa3","Type":"ContainerStarted","Data":"d339ac3f310437ea8b0a27211ee021798a563cf7d7dc2cddd5c02f15713f1535"} Jul 10 13:10:55.272016 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.271991 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" event={"ID":"79709752-cad4-43ff-b5a8-3575462085a3","Type":"ContainerStarted","Data":"515f7436e8aebdb2cb5f9cfbefd4c676af52e3ccacaeb5256bf5d5f69202a9f1"} Jul 10 13:10:55.305719 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.305679 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gq7bz" podStartSLOduration=2.4254334699999998 podStartE2EDuration="4.305664541s" totalImagesPullingTime="1.880231071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:55.304188683 +0000 UTC m=+52.836790981" watchObservedRunningTime="2026-07-10 13:10:55.305664541 +0000 UTC m=+52.838266839" Jul 10 13:10:55.337956 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:55.337908 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-22qdg" podStartSLOduration=2.337890158 podStartE2EDuration="2.337890158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:55.336911352 +0000 UTC m=+52.869513656" watchObservedRunningTime="2026-07-10 13:10:55.337890158 +0000 UTC m=+52.870492456" Jul 10 13:10:56.212716 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.212683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:56.212716 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.212723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.212745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.212858 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.212863 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.212858 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.212839057 +0000 UTC m=+69.745441357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : configmap references non-existent config key: service-ca.crt Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.212901 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls podName:ed580227-8d8f-49ad-b10f-4d138f1c8e2c nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.212891481 +0000 UTC m=+69.745493765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls") pod "cluster-samples-operator-65db858894-j8pj5" (UID: "ed580227-8d8f-49ad-b10f-4d138f1c8e2c") : secret "samples-operator-tls" not found Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.212924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs podName:ca3745bd-546f-4cc6-8edf-672d5511f623 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.212906381 +0000 UTC m=+69.745508670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs") pod "router-default-755b7d5488-4vbsm" (UID: "ca3745bd-546f-4cc6-8edf-672d5511f623") : secret "router-metrics-certs-default" not found Jul 10 13:10:56.212979 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.212950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.213003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.213067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213105 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213125 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655875787f-6ht6h: secret "image-registry-tls" not found Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:56.213145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213160 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls podName:65e7898d-2f6b-42d5-83a4-eb23f29bc2c6 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.213148987 +0000 UTC m=+69.745751287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls") pod "image-registry-655875787f-6ht6h" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6") : secret "image-registry-tls" not found Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213184 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 13:10:56.213222 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213220 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:56.213464 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213228 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 10 13:10:56.213464 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213239 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls podName:afc44162-1c8b-427f-a7fb-ac64c02eb3af nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.213218195 +0000 UTC m=+69.745820474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls") pod "dns-default-jjh5l" (UID: "afc44162-1c8b-427f-a7fb-ac64c02eb3af") : secret "dns-default-metrics-tls" not found Jul 10 13:10:56.213464 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213258 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls podName:d6a98def-23d5-404f-8429-18347a24e8b9 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.2132453 +0000 UTC m=+69.745847586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-jxjvk" (UID: "d6a98def-23d5-404f-8429-18347a24e8b9") : secret "cluster-monitoring-operator-tls" not found Jul 10 13:10:56.213464 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:56.213273 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert podName:b364a518-6d8e-4b67-b242-d64942c29b50 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:12.213264595 +0000 UTC m=+69.745866878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-qk7gx" (UID: "b364a518-6d8e-4b67-b242-d64942c29b50") : secret "networking-console-plugin-cert" not found Jul 10 13:10:59.141512 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:59.141477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:59.141893 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:10:59.141687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:10:59.141893 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:59.141776 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 13:10:59.141893 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:59.141862 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:07.141839675 +0000 UTC m=+64.674441951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : configmap "kube-rbac-proxy" not found Jul 10 13:10:59.142072 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:59.142051 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 13:10:59.142125 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:10:59.142115 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls podName:44f5d802-1f73-4843-ac3e-c7985f017eb6 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:07.142099168 +0000 UTC m=+64.674701446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmhkk" (UID: "44f5d802-1f73-4843-ac3e-c7985f017eb6") : secret "insights-runtime-extractor-tls" not found Jul 10 13:11:00.680025 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:00.679985 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:11:00.680588 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:00.680077 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:11:00.680588 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:00.680440 2573 scope.go:117] "RemoveContainer" containerID="b0141ad0576aa5f54dedab921d19392ece748453e46105c79275ac6644930724" Jul 10 13:11:01.296304 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:01.296276 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:11:01.296455 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:01.296349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jdskh" event={"ID":"509c9cde-d447-4dd4-b153-37eb06725a6a","Type":"ContainerStarted","Data":"76a9a9e04dc1fb686323937a339f41624927a85526c57c2d7a696275d4a5411f"} Jul 10 13:11:01.296698 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:01.296677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:11:01.302771 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:01.302749 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-jdskh" Jul 10 13:11:01.316213 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:01.316170 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-jdskh" podStartSLOduration=45.905590852 podStartE2EDuration="53.316156708s" totalImagesPullingTime="7.410565856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:01.315456774 +0000 UTC m=+58.848059089" watchObservedRunningTime="2026-07-10 13:11:01.316156708 +0000 UTC m=+58.848759007" Jul 10 13:11:04.143474 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:04.143445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjbtb" Jul 10 13:11:07.213506 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.213463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:11:07.213905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.213543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:11:07.214023 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.214002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44f5d802-1f73-4843-ac3e-c7985f017eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:11:07.215783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.215768 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44f5d802-1f73-4843-ac3e-c7985f017eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmhkk\" (UID: \"44f5d802-1f73-4843-ac3e-c7985f017eb6\") " pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:11:07.320169 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.320144 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4zgls\"" Jul 10 13:11:07.328253 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.328233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmhkk" Jul 10 13:11:07.450143 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:07.450118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmhkk"] Jul 10 13:11:07.452558 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:07.452531 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f5d802_1f73_4843_ac3e_c7985f017eb6.slice/crio-90d9e797a4cc1eee8a58109ab60b945713d3fd4a01e65d0c7520d123c6ee6cbe WatchSource:0}: Error finding container 90d9e797a4cc1eee8a58109ab60b945713d3fd4a01e65d0c7520d123c6ee6cbe: Status 404 returned error can't find the container with id 90d9e797a4cc1eee8a58109ab60b945713d3fd4a01e65d0c7520d123c6ee6cbe Jul 10 13:11:08.325326 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.325289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmhkk" event={"ID":"44f5d802-1f73-4843-ac3e-c7985f017eb6","Type":"ContainerStarted","Data":"90d9e797a4cc1eee8a58109ab60b945713d3fd4a01e65d0c7520d123c6ee6cbe"} Jul 10 13:11:08.627960 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.627926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:11:08.628142 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.627988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:11:08.630599 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.630574 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:11:08.630737 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.630599 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:11:08.641054 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.641032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36ab361d-d9c8-4d97-bf03-4393b79c32f8-original-pull-secret\") pod \"global-pull-secret-syncer-28kx4\" (UID: \"36ab361d-d9c8-4d97-bf03-4393b79c32f8\") " pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:11:08.641054 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.641042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbcdad2-6da8-4797-b9c0-f07a6964baf9-metrics-certs\") pod \"network-metrics-daemon-t95jr\" (UID: \"7bbcdad2-6da8-4797-b9c0-f07a6964baf9\") " pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:11:08.728560 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.728529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:11:08.731110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.731084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mb6\" (UniqueName: \"kubernetes.io/projected/75bfff61-c6ee-4ccc-96a8-a314e2b35abf-kube-api-access-t8mb6\") pod \"network-check-target-nmbs2\" (UID: \"75bfff61-c6ee-4ccc-96a8-a314e2b35abf\") " pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:11:08.877393 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.877362 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xcvz\"" Jul 10 13:11:08.883743 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.883625 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnrp9\"" Jul 10 13:11:08.885441 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.885419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t95jr" Jul 10 13:11:08.886588 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.886559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-28kx4" Jul 10 13:11:08.892051 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:08.892030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:11:09.049510 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:09.049479 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t95jr"] Jul 10 13:11:09.268705 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:09.268683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-28kx4"] Jul 10 13:11:09.271437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:09.271413 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nmbs2"] Jul 10 13:11:10.227083 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:10.227046 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbcdad2_6da8_4797_b9c0_f07a6964baf9.slice/crio-d5f07eacdf28078c14308f414a7982e7460cc8fb41fc33f8dd71030925c3551f WatchSource:0}: Error finding container d5f07eacdf28078c14308f414a7982e7460cc8fb41fc33f8dd71030925c3551f: Status 404 returned error can't find the container with id d5f07eacdf28078c14308f414a7982e7460cc8fb41fc33f8dd71030925c3551f Jul 10 13:11:10.227509 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:10.227465 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bfff61_c6ee_4ccc_96a8_a314e2b35abf.slice/crio-ed95b121a5043902867f23740d5f924b9e1a5e2725dd7443785b3bdd6fc6dee5 WatchSource:0}: Error finding container ed95b121a5043902867f23740d5f924b9e1a5e2725dd7443785b3bdd6fc6dee5: Status 404 returned error can't find the container with id ed95b121a5043902867f23740d5f924b9e1a5e2725dd7443785b3bdd6fc6dee5 Jul 10 13:11:10.264368 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:10.264305 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab361d_d9c8_4d97_bf03_4393b79c32f8.slice/crio-42257be72887d44342faee41768495531dfb97ed51b656ba5278207f3aa68542 WatchSource:0}: Error finding container 42257be72887d44342faee41768495531dfb97ed51b656ba5278207f3aa68542: Status 404 returned error can't find the container with id 42257be72887d44342faee41768495531dfb97ed51b656ba5278207f3aa68542 Jul 10 13:11:10.333704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:10.333662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nmbs2" event={"ID":"75bfff61-c6ee-4ccc-96a8-a314e2b35abf","Type":"ContainerStarted","Data":"ed95b121a5043902867f23740d5f924b9e1a5e2725dd7443785b3bdd6fc6dee5"} Jul 10 13:11:10.335059 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:10.335033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-28kx4" event={"ID":"36ab361d-d9c8-4d97-bf03-4393b79c32f8","Type":"ContainerStarted","Data":"42257be72887d44342faee41768495531dfb97ed51b656ba5278207f3aa68542"} Jul 10 13:11:10.336338 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:10.336314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t95jr" event={"ID":"7bbcdad2-6da8-4797-b9c0-f07a6964baf9","Type":"ContainerStarted","Data":"d5f07eacdf28078c14308f414a7982e7460cc8fb41fc33f8dd71030925c3551f"} Jul 10 13:11:11.342440 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:11.342397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nmbs2" event={"ID":"75bfff61-c6ee-4ccc-96a8-a314e2b35abf","Type":"ContainerStarted","Data":"fc06265fb7338cef2c8fd150916464e5e397e37264f4e89af9da7538e5fdd91a"} Jul 10 13:11:11.342895 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:11.342571 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:11:11.363387 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:11.362848 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nmbs2" podStartSLOduration=68.362830627 podStartE2EDuration="1m8.362830627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:11.362159628 +0000 UTC m=+68.894761928" watchObservedRunningTime="2026-07-10 13:11:11.362830627 +0000 UTC m=+68.895432927" Jul 10 13:11:12.260683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.260625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:11:12.260796 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.260716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.261115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.261211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.261258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.261285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.261331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:11:12.263420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.262242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca3745bd-546f-4cc6-8edf-672d5511f623-service-ca-bundle\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.264260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afc44162-1c8b-427f-a7fb-ac64c02eb3af-metrics-tls\") pod \"dns-default-jjh5l\" (UID: \"afc44162-1c8b-427f-a7fb-ac64c02eb3af\") " pod="openshift-dns/dns-default-jjh5l" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.265126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6a98def-23d5-404f-8429-18347a24e8b9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-jxjvk\" (UID: \"d6a98def-23d5-404f-8429-18347a24e8b9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.265130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b364a518-6d8e-4b67-b242-d64942c29b50-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qk7gx\" (UID: \"b364a518-6d8e-4b67-b242-d64942c29b50\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.265354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"image-registry-655875787f-6ht6h\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.268466 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed580227-8d8f-49ad-b10f-4d138f1c8e2c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-j8pj5\" (UID: \"ed580227-8d8f-49ad-b10f-4d138f1c8e2c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:11:12.268614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.268584 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3745bd-546f-4cc6-8edf-672d5511f623-metrics-certs\") pod \"router-default-755b7d5488-4vbsm\" (UID: \"ca3745bd-546f-4cc6-8edf-672d5511f623\") " pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:12.348186 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.348156 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmhkk" event={"ID":"44f5d802-1f73-4843-ac3e-c7985f017eb6","Type":"ContainerStarted","Data":"59c9272e938886d17b802a6c8fe88e178ae295bc1e832c3de3530caa028cd3fc"} Jul 10 13:11:12.348561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.348253 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmhkk" event={"ID":"44f5d802-1f73-4843-ac3e-c7985f017eb6","Type":"ContainerStarted","Data":"a00e9116355d065f7bd6f958f93f29acd330b766d237786d9ce9b2d9e0805dde"} Jul 10 13:11:12.349803 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.349775 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t95jr" event={"ID":"7bbcdad2-6da8-4797-b9c0-f07a6964baf9","Type":"ContainerStarted","Data":"4327c8d4c8f4505afea4bcfe99b34030141c94c618cfc06dead8242fd4b52ca1"} Jul 10 13:11:12.349898 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.349825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t95jr" event={"ID":"7bbcdad2-6da8-4797-b9c0-f07a6964baf9","Type":"ContainerStarted","Data":"8b59a7a03cd08ec32fa6a9afe056bae109e7661c14d520fbea08d2c3c39dd0d7"} Jul 10 13:11:12.368728 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.368683 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t95jr" podStartSLOduration=68.110833509 podStartE2EDuration="1m9.368668375s" totalImagesPullingTime="1.257834866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:12.368280195 +0000 UTC m=+69.900882498" watchObservedRunningTime="2026-07-10 13:11:12.368668375 +0000 UTC m=+69.901270674" Jul 10 13:11:12.433743 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.433717 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ngc6m\"" Jul 10 13:11:12.441606 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.441548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:11:12.460166 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.460142 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-75574\"" Jul 10 13:11:12.468645 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.468606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" Jul 10 13:11:12.472052 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.472025 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wvglq\"" Jul 10 13:11:12.479848 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.479827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" Jul 10 13:11:12.494303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.493256 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h8rcs\"" Jul 10 13:11:12.497968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.496899 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7p5lh\"" Jul 10 13:11:12.501535 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.501509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:12.505450 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.505365 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" Jul 10 13:11:12.531022 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.530992 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vrvn5\"" Jul 10 13:11:12.539890 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.539448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjh5l" Jul 10 13:11:12.657935 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.656943 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:11:12.696520 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.696463 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk"] Jul 10 13:11:12.704828 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:12.704795 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a98def_23d5_404f_8429_18347a24e8b9.slice/crio-65e91ae89fcb779a75caeb3f19c55647697671fc3c400b2eafaff3c213a9ae02 WatchSource:0}: Error finding container 65e91ae89fcb779a75caeb3f19c55647697671fc3c400b2eafaff3c213a9ae02: Status 404 returned error can't find the container with id 65e91ae89fcb779a75caeb3f19c55647697671fc3c400b2eafaff3c213a9ae02 Jul 10 13:11:12.718932 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.718671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5"] Jul 10 13:11:12.743221 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.743137 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-755b7d5488-4vbsm"] Jul 10 13:11:12.770074 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.770044 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qk7gx"] Jul 10 13:11:12.774268 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:12.774232 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364a518_6d8e_4b67_b242_d64942c29b50.slice/crio-19b8efe1a657f0d05dd8798641273b65685eb8bc6eac5ec8c46c57653cbc283d WatchSource:0}: Error finding container 19b8efe1a657f0d05dd8798641273b65685eb8bc6eac5ec8c46c57653cbc283d: Status 404 returned error can't find the container with id 19b8efe1a657f0d05dd8798641273b65685eb8bc6eac5ec8c46c57653cbc283d Jul 10 13:11:12.804888 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:12.803826 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjh5l"] Jul 10 13:11:12.810143 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:12.810118 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc44162_1c8b_427f_a7fb_ac64c02eb3af.slice/crio-e5dbbf69718c35addce78e7aebd1543220b9aa059131786500b861fa6319bf3d WatchSource:0}: Error finding container e5dbbf69718c35addce78e7aebd1543220b9aa059131786500b861fa6319bf3d: Status 404 returned error can't find the container with id e5dbbf69718c35addce78e7aebd1543220b9aa059131786500b861fa6319bf3d Jul 10 13:11:13.356034 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.355989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjh5l" event={"ID":"afc44162-1c8b-427f-a7fb-ac64c02eb3af","Type":"ContainerStarted","Data":"e5dbbf69718c35addce78e7aebd1543220b9aa059131786500b861fa6319bf3d"} Jul 10 13:11:13.357483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.357425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" event={"ID":"ed580227-8d8f-49ad-b10f-4d138f1c8e2c","Type":"ContainerStarted","Data":"248af60eb8ec0ed267a94a82b8176a5581739b8ab7ec3a409bf5a7773f9d91d6"} Jul 10 13:11:13.358896 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.358871 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" event={"ID":"b364a518-6d8e-4b67-b242-d64942c29b50","Type":"ContainerStarted","Data":"19b8efe1a657f0d05dd8798641273b65685eb8bc6eac5ec8c46c57653cbc283d"} Jul 10 13:11:13.360473 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.360449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655875787f-6ht6h" event={"ID":"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6","Type":"ContainerStarted","Data":"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06"} Jul 10 13:11:13.360570 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.360491 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655875787f-6ht6h" event={"ID":"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6","Type":"ContainerStarted","Data":"61ddabb59f42a2e31eb01b87a9a2960161797ee7b851c1f0335c7ec6201f3801"} Jul 10 13:11:13.360570 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.360526 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:11:13.362218 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.362194 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-755b7d5488-4vbsm" event={"ID":"ca3745bd-546f-4cc6-8edf-672d5511f623","Type":"ContainerStarted","Data":"f3ff0200b8fbd7d75546e7c17b33f4dced7308764671c681290f07fe6ba9a05e"} Jul 10 13:11:13.362302 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.362243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-755b7d5488-4vbsm" event={"ID":"ca3745bd-546f-4cc6-8edf-672d5511f623","Type":"ContainerStarted","Data":"d68f8d155f43c730742f9b1bb977cacae0bbe71b7c203b148668e17e39271942"} Jul 10 13:11:13.363656 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.363607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" event={"ID":"d6a98def-23d5-404f-8429-18347a24e8b9","Type":"ContainerStarted","Data":"65e91ae89fcb779a75caeb3f19c55647697671fc3c400b2eafaff3c213a9ae02"} Jul 10 13:11:13.381462 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.381416 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-655875787f-6ht6h" podStartSLOduration=69.381398 podStartE2EDuration="1m9.381398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:13.37960862 +0000 UTC m=+70.912210916" watchObservedRunningTime="2026-07-10 13:11:13.381398 +0000 UTC m=+70.914000299" Jul 10 13:11:13.398378 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.398331 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-755b7d5488-4vbsm" podStartSLOduration=65.398314654 podStartE2EDuration="1m5.398314654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:13.39623564 +0000 UTC m=+70.928837939" watchObservedRunningTime="2026-07-10 13:11:13.398314654 +0000 UTC m=+70.930916955" Jul 10 13:11:13.502186 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.502157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:13.505290 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:13.505108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:14.369045 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:14.369007 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:14.370499 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:14.370473 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-755b7d5488-4vbsm" Jul 10 13:11:17.606017 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.605983 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v"] Jul 10 13:11:17.685769 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.685727 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84"] Jul 10 13:11:17.685924 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.685872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.692428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.691029 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 13:11:17.692428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.691287 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 13:11:17.692428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.692190 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 13:11:17.692428 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.692227 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 13:11:17.720757 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.720721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84"] Jul 10 13:11:17.720891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.720768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm"] Jul 10 13:11:17.720891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.720828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.726811 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.726792 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 13:11:17.728529 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.728355 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 13:11:17.728529 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.728379 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 13:11:17.728529 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.728515 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 13:11:17.743278 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.743255 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v"] Jul 10 13:11:17.743429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.743301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm"] Jul 10 13:11:17.743429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.743331 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-tlxnd"] Jul 10 13:11:17.743429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.743370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:17.746019 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.745999 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lncdj\"" Jul 10 13:11:17.746394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.746377 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 13:11:17.759168 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.759147 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:11:17.759301 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.759285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:17.761888 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.761871 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 13:11:17.762316 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.762298 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 13:11:17.762316 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.762309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s7bzh\"" Jul 10 13:11:17.785209 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.785182 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tlxnd"] Jul 10 13:11:17.785304 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.785222 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:11:17.785358 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.785311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.790010 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.789991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 13:11:17.790106 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.790082 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 13:11:17.790321 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.790303 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vszsp\"" Jul 10 13:11:17.790391 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.790326 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 13:11:17.790391 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.790378 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 13:11:17.790461 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.790332 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 13:11:17.811858 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.811834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-tmp\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.812136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.811985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.812136 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.812029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7x5\" (UniqueName: \"kubernetes.io/projected/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-kube-api-access-hv7x5\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.912700 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.912875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.912875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7x5\" (UniqueName: \"kubernetes.io/projected/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-kube-api-access-hv7x5\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.912875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfhf\" (UniqueName: \"kubernetes.io/projected/228f3243-96f6-49a8-935a-6a7bf9d04c84-kube-api-access-vgfhf\") pod \"downloads-68c9cbb497-tlxnd\" (UID: \"228f3243-96f6-49a8-935a-6a7bf9d04c84\") " pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:17.912875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnrm\" (UniqueName: \"kubernetes.io/projected/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-kube-api-access-pfnrm\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:17.913079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8td6r\" (UniqueName: \"kubernetes.io/projected/812a807f-fe7b-433b-9b4b-8eb71a782fe4-kube-api-access-8td6r\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-tmp\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.913079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.912966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.913079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.913079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpj8\" (UniqueName: \"kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/812a807f-fe7b-433b-9b4b-8eb71a782fe4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.913309 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-tmp\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.913595 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:17.913595 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.913366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:17.916079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.916055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:17.924169 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:17.924145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7x5\" (UniqueName: \"kubernetes.io/projected/e83a5afb-41d4-4d00-a3d6-4d6623b5b758-kube-api-access-hv7x5\") pod \"klusterlet-addon-workmgr-7b6b97c8d8-zrl9v\" (UID: \"e83a5afb-41d4-4d00-a3d6-4d6623b5b758\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:18.003219 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.003192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:18.014046 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014141 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/812a807f-fe7b-433b-9b4b-8eb71a782fe4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014141 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014255 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:18.014306 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014306 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfhf\" (UniqueName: \"kubernetes.io/projected/228f3243-96f6-49a8-935a-6a7bf9d04c84-kube-api-access-vgfhf\") pod \"downloads-68c9cbb497-tlxnd\" (UID: \"228f3243-96f6-49a8-935a-6a7bf9d04c84\") " pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:18.014402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnrm\" (UniqueName: \"kubernetes.io/projected/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-kube-api-access-pfnrm\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:18.014402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8td6r\" (UniqueName: \"kubernetes.io/projected/812a807f-fe7b-433b-9b4b-8eb71a782fe4-kube-api-access-8td6r\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpj8\" (UniqueName: \"kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014781 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.014781 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.014781 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.014746 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/812a807f-fe7b-433b-9b4b-8eb71a782fe4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.015429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.015403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.017314 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-ca\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.017314 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.017314 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.017314 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.017593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017341 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.017593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017366 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:18.017820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.017796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.018028 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.018009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.018551 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.018535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/812a807f-fe7b-433b-9b4b-8eb71a782fe4-hub\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.025239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.025215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnrm\" (UniqueName: \"kubernetes.io/projected/a6b50826-abaf-4bc1-8cb5-f64af9ec4e09-kube-api-access-pfnrm\") pod \"managed-serviceaccount-addon-agent-7c584b8f5d-d55pm\" (UID: \"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:18.027871 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.027838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8td6r\" (UniqueName: \"kubernetes.io/projected/812a807f-fe7b-433b-9b4b-8eb71a782fe4-kube-api-access-8td6r\") pod \"cluster-proxy-proxy-agent-76b758558d-r8w84\" (UID: \"812a807f-fe7b-433b-9b4b-8eb71a782fe4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.027996 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.027980 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfhf\" (UniqueName: \"kubernetes.io/projected/228f3243-96f6-49a8-935a-6a7bf9d04c84-kube-api-access-vgfhf\") pod \"downloads-68c9cbb497-tlxnd\" (UID: \"228f3243-96f6-49a8-935a-6a7bf9d04c84\") " pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:18.028294 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.028278 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpj8\" (UniqueName: \"kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8\") pod \"console-67dcffcf6c-6blgh\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:18.040118 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.040100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" Jul 10 13:11:18.053161 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.053141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" Jul 10 13:11:18.084886 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.084860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:18.096972 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:18.096951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:20.111222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.111193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:11:20.122146 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:20.122013 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda020c31a_61f8_45e7_b698_697e5a9d3c87.slice/crio-9e7802ed2fd3311a7c94e805b5e3c97880a2ea8f71f0a957d84e4520ace7b6c9 WatchSource:0}: Error finding container 9e7802ed2fd3311a7c94e805b5e3c97880a2ea8f71f0a957d84e4520ace7b6c9: Status 404 returned error can't find the container with id 9e7802ed2fd3311a7c94e805b5e3c97880a2ea8f71f0a957d84e4520ace7b6c9 Jul 10 13:11:20.164324 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.164267 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tlxnd"] Jul 10 13:11:20.175861 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:20.175830 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228f3243_96f6_49a8_935a_6a7bf9d04c84.slice/crio-459c83056655ec569a5832f000c5c32f2e6f17a685f7e904eec684ae3bfdacef WatchSource:0}: Error finding container 459c83056655ec569a5832f000c5c32f2e6f17a685f7e904eec684ae3bfdacef: Status 404 returned error can't find the container with id 459c83056655ec569a5832f000c5c32f2e6f17a685f7e904eec684ae3bfdacef Jul 10 13:11:20.383168 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.383138 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm"] Jul 10 13:11:20.386017 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.385930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v"] Jul 10 13:11:20.389925 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.389900 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84"] Jul 10 13:11:20.406449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.406419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmhkk" event={"ID":"44f5d802-1f73-4843-ac3e-c7985f017eb6","Type":"ContainerStarted","Data":"1e2c564f561f1c53301832e227cb9ad24c15dd08ac3b0defd3cddc7f1174e56f"} Jul 10 13:11:20.408763 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.408737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" event={"ID":"d6a98def-23d5-404f-8429-18347a24e8b9","Type":"ContainerStarted","Data":"7198e2cd4c317ac5f7db1cf7afb0b019defba187e8ddf516520e723b040bfc6a"} Jul 10 13:11:20.410973 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.410929 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-28kx4" event={"ID":"36ab361d-d9c8-4d97-bf03-4393b79c32f8","Type":"ContainerStarted","Data":"43744e358bbc35d4167445e5b82be7a669a88f5cfeb1011a575a0ba989b23b1b"} Jul 10 13:11:20.412923 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.412898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjh5l" event={"ID":"afc44162-1c8b-427f-a7fb-ac64c02eb3af","Type":"ContainerStarted","Data":"ab4ef8dd00bfb8345baf60fe6f0218ae5cf45fea193de09c878442eb88a9f19f"} Jul 10 13:11:20.414822 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.414797 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" event={"ID":"ed580227-8d8f-49ad-b10f-4d138f1c8e2c","Type":"ContainerStarted","Data":"7d3e18c8406f0094a68ef0d5aa3ec5d6ebc79d885d8b83f5a7e6489703a7b83d"} Jul 10 13:11:20.414912 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.414877 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" event={"ID":"ed580227-8d8f-49ad-b10f-4d138f1c8e2c","Type":"ContainerStarted","Data":"b1ce8eea297bf2d95bd40d0c56c840426acef3250a3e6dedf74b4be3122b99a0"} Jul 10 13:11:20.416553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.416530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" event={"ID":"b364a518-6d8e-4b67-b242-d64942c29b50","Type":"ContainerStarted","Data":"7e2e1eb8fc6de6daf5a1437f18ef19210957aba045315702a6ba3fa48a5a706f"} Jul 10 13:11:20.418118 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.418096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tlxnd" event={"ID":"228f3243-96f6-49a8-935a-6a7bf9d04c84","Type":"ContainerStarted","Data":"459c83056655ec569a5832f000c5c32f2e6f17a685f7e904eec684ae3bfdacef"} Jul 10 13:11:20.419288 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.419258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dcffcf6c-6blgh" event={"ID":"a020c31a-61f8-45e7-b698-697e5a9d3c87","Type":"ContainerStarted","Data":"9e7802ed2fd3311a7c94e805b5e3c97880a2ea8f71f0a957d84e4520ace7b6c9"} Jul 10 13:11:20.436227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.436165 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vmhkk" podStartSLOduration=17.170699017 podStartE2EDuration="29.436146451s" totalImagesPullingTime="12.265447434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:20.434493168 +0000 UTC m=+77.967095471" watchObservedRunningTime="2026-07-10 13:11:20.436146451 +0000 UTC m=+77.968748751" Jul 10 13:11:20.458457 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.458414 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-jxjvk" podStartSLOduration=65.265730112 podStartE2EDuration="1m12.458379565s" totalImagesPullingTime="7.192649453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:20.456285487 +0000 UTC m=+77.988887786" watchObservedRunningTime="2026-07-10 13:11:20.458379565 +0000 UTC m=+77.990981867" Jul 10 13:11:20.478690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.478621 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-qk7gx" podStartSLOduration=65.350747477 podStartE2EDuration="1m12.478600609s" totalImagesPullingTime="7.127853132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:20.477068354 +0000 UTC m=+78.009670678" watchObservedRunningTime="2026-07-10 13:11:20.478600609 +0000 UTC m=+78.011202909" Jul 10 13:11:20.503746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.503677 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-j8pj5" podStartSLOduration=65.371233127 podStartE2EDuration="1m12.503665261s" totalImagesPullingTime="7.132432134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:20.503213746 +0000 UTC m=+78.035816046" watchObservedRunningTime="2026-07-10 13:11:20.503665261 +0000 UTC m=+78.036267554" Jul 10 13:11:20.539905 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:20.539860 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-28kx4" podStartSLOduration=67.900691983 podStartE2EDuration="1m17.539834085s" totalImagesPullingTime="9.639142102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:20.539169292 +0000 UTC m=+78.071771591" watchObservedRunningTime="2026-07-10 13:11:20.539834085 +0000 UTC m=+78.072436378" Jul 10 13:11:21.429970 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.429912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjh5l" event={"ID":"afc44162-1c8b-427f-a7fb-ac64c02eb3af","Type":"ContainerStarted","Data":"8e2359dbe6dae154fe07439ef9e3a4dc9be6ea2503d77da1961e792cf1594cf8"} Jul 10 13:11:21.430868 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.430804 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjh5l" Jul 10 13:11:21.435739 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.435693 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" event={"ID":"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09","Type":"ContainerStarted","Data":"dbf692b2ef428042310f5aadae76091b8f64e551ec370eea2f906ba6ba5a0633"} Jul 10 13:11:21.437659 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.437586 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" event={"ID":"e83a5afb-41d4-4d00-a3d6-4d6623b5b758","Type":"ContainerStarted","Data":"6e6807fa7872123b91678f23eda4e47792add0a679c2b8823d73617ab3b4f0a2"} Jul 10 13:11:21.443331 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.440413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" event={"ID":"812a807f-fe7b-433b-9b4b-8eb71a782fe4","Type":"ContainerStarted","Data":"df172e8463a41d517a7b651787bea05b3fd8f3e946bd129d39b71df48cc35ede"} Jul 10 13:11:21.453643 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:21.453583 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jjh5l" podStartSLOduration=34.358313425 podStartE2EDuration="41.453565136s" totalImagesPullingTime="7.095251711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:21.451780675 +0000 UTC m=+78.984382975" watchObservedRunningTime="2026-07-10 13:11:21.453565136 +0000 UTC m=+78.986167437" Jul 10 13:11:29.489998 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.489954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dcffcf6c-6blgh" event={"ID":"a020c31a-61f8-45e7-b698-697e5a9d3c87","Type":"ContainerStarted","Data":"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13"} Jul 10 13:11:29.491806 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.491774 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" event={"ID":"a6b50826-abaf-4bc1-8cb5-f64af9ec4e09","Type":"ContainerStarted","Data":"3f03ec0efb8f8d592fb10047ac6e9dd14f9705eae0abf0366adb98db83c64d5a"} Jul 10 13:11:29.493322 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.493296 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" event={"ID":"e83a5afb-41d4-4d00-a3d6-4d6623b5b758","Type":"ContainerStarted","Data":"b0ae6a26942221501ef0415f597f0f993a70fb10ebcab50cdca767f5443bd6d4"} Jul 10 13:11:29.493484 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.493461 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:29.495303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.495262 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" Jul 10 13:11:29.496159 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.496134 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" event={"ID":"812a807f-fe7b-433b-9b4b-8eb71a782fe4","Type":"ContainerStarted","Data":"693d5ebf0ce3bb791f7abd10c6756b7f9f03a3b9f3d2aeed3e8f20efd61f05c9"} Jul 10 13:11:29.496250 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.496177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" event={"ID":"812a807f-fe7b-433b-9b4b-8eb71a782fe4","Type":"ContainerStarted","Data":"b9fdfec909806034ac2221597f83b885c758286a13f8dc45905d2e7e1e8d2b9b"} Jul 10 13:11:29.496250 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.496195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" event={"ID":"812a807f-fe7b-433b-9b4b-8eb71a782fe4","Type":"ContainerStarted","Data":"05ad257e23c4a22d282ce1c94c88fe31d8adac047aa2144b72a6b40d638058b8"} Jul 10 13:11:29.511802 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.511762 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67dcffcf6c-6blgh" podStartSLOduration=3.97764881 podStartE2EDuration="12.511747059s" totalImagesPullingTime="8.534098249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:29.510446696 +0000 UTC m=+87.043049019" watchObservedRunningTime="2026-07-10 13:11:29.511747059 +0000 UTC m=+87.044349357" Jul 10 13:11:29.528723 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.528652 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6b97c8d8-zrl9v" podStartSLOduration=4.24566677 podStartE2EDuration="12.528620956s" totalImagesPullingTime="8.282954186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:29.527575748 +0000 UTC m=+87.060178048" watchObservedRunningTime="2026-07-10 13:11:29.528620956 +0000 UTC m=+87.061223256" Jul 10 13:11:29.551053 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:29.551012 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b758558d-r8w84" podStartSLOduration=4.2774600320000005 podStartE2EDuration="12.550998492s" totalImagesPullingTime="8.27353846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:29.548660934 +0000 UTC m=+87.081263230" watchObservedRunningTime="2026-07-10 13:11:29.550998492 +0000 UTC m=+87.083600790" Jul 10 13:11:32.446129 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:32.446089 2573 patch_prober.go:28] interesting pod/image-registry-655875787f-6ht6h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 13:11:32.446492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:32.446147 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-655875787f-6ht6h" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:11:32.458679 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:32.458625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjh5l" Jul 10 13:11:32.487661 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:32.487603 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c584b8f5d-d55pm" podStartSLOduration=7.222432408 podStartE2EDuration="15.48759085s" totalImagesPullingTime="8.265158442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:29.569963042 +0000 UTC m=+87.102565342" watchObservedRunningTime="2026-07-10 13:11:32.48759085 +0000 UTC m=+90.020193149" Jul 10 13:11:34.374108 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.374080 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:11:34.694019 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.693894 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pcqmr"] Jul 10 13:11:34.697946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.697923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.709085 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.708999 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 13:11:34.711687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.711591 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 13:11:34.711687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.711657 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 13:11:34.712125 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.711982 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6hvq\"" Jul 10 13:11:34.712125 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.712104 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 13:11:34.761400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-sys\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkg8w\" (UniqueName: \"kubernetes.io/projected/642eb80f-115d-4e22-9761-b44da23c69af-kube-api-access-bkg8w\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-root\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-textfile\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761752 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-wtmp\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761752 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-metrics-client-ca\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761752 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.761752 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.761688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-tls\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862591 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-wtmp\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862777 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-metrics-client-ca\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862777 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862777 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-tls\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862777 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-wtmp\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862998 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-sys\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862998 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkg8w\" (UniqueName: \"kubernetes.io/projected/642eb80f-115d-4e22-9761-b44da23c69af-kube-api-access-bkg8w\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.862998 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.862858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863253 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-metrics-client-ca\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863336 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863282 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-sys\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-root\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-textfile\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863487 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863487 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642eb80f-115d-4e22-9761-b44da23c69af-root\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.863746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.863703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-textfile\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.865735 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.865705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-tls\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.865837 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.865769 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642eb80f-115d-4e22-9761-b44da23c69af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:34.915798 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:34.915767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkg8w\" (UniqueName: \"kubernetes.io/projected/642eb80f-115d-4e22-9761-b44da23c69af-kube-api-access-bkg8w\") pod \"node-exporter-pcqmr\" (UID: \"642eb80f-115d-4e22-9761-b44da23c69af\") " pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:35.011157 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:35.011084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcqmr" Jul 10 13:11:35.023845 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:35.023812 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642eb80f_115d_4e22_9761_b44da23c69af.slice/crio-ea90c75bab498f57b1d6ce856e04dfed35a4df7f2e0faf276f2a8a1db47f8917 WatchSource:0}: Error finding container ea90c75bab498f57b1d6ce856e04dfed35a4df7f2e0faf276f2a8a1db47f8917: Status 404 returned error can't find the container with id ea90c75bab498f57b1d6ce856e04dfed35a4df7f2e0faf276f2a8a1db47f8917 Jul 10 13:11:35.530020 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:35.529912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcqmr" event={"ID":"642eb80f-115d-4e22-9761-b44da23c69af","Type":"ContainerStarted","Data":"ea90c75bab498f57b1d6ce856e04dfed35a4df7f2e0faf276f2a8a1db47f8917"} Jul 10 13:11:36.535727 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:36.535695 2573 generic.go:356] "Generic (PLEG): container finished" podID="642eb80f-115d-4e22-9761-b44da23c69af" containerID="e725b79448bca70d01d416e482ba9f95b14ad830ffd6e50cacd88d28be7fea9b" exitCode=0 Jul 10 13:11:36.536031 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:36.535772 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcqmr" event={"ID":"642eb80f-115d-4e22-9761-b44da23c69af","Type":"ContainerDied","Data":"e725b79448bca70d01d416e482ba9f95b14ad830ffd6e50cacd88d28be7fea9b"} Jul 10 13:11:37.543399 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:37.543363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcqmr" event={"ID":"642eb80f-115d-4e22-9761-b44da23c69af","Type":"ContainerStarted","Data":"a6b2c68f4e1fe329bf58cfd7479a85e40ae32f3bb3aba3ea87ea519866ddc99d"} Jul 10 13:11:37.543843 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:37.543427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcqmr" event={"ID":"642eb80f-115d-4e22-9761-b44da23c69af","Type":"ContainerStarted","Data":"15f6ef8f6080622e0614f1de313bc65314190b1767ee9f0827a796e91a735bac"} Jul 10 13:11:37.584229 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:37.584174 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pcqmr" podStartSLOduration=2.176590328 podStartE2EDuration="3.58415546s" totalImagesPullingTime="1.342256471s" totalInitContainerRuntime="65.308661ms" isStatefulPod=true podCreationTimestamp="2026-07-10 13:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:37.583333821 +0000 UTC m=+95.115936119" watchObservedRunningTime="2026-07-10 13:11:37.58415546 +0000 UTC m=+95.116757760" Jul 10 13:11:38.097366 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:38.097331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:38.097547 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:38.097418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:38.102417 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:38.102390 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:38.552730 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:38.552700 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:11:40.736156 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:40.736123 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:11:40.905847 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:40.905803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:11:40.906003 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:40.905823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:40.915824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:40.915798 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 13:11:41.014338 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zjn\" (UniqueName: \"kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014338 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014538 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014538 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.014683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.014615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115649 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zjn\" (UniqueName: \"kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115985 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.115985 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.116089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.115991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.116475 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.116380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.116575 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.116479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.116652 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.116596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.116757 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.116736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.118948 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.118922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.119034 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.118966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.133402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.133382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zjn\" (UniqueName: \"kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn\") pod \"console-977766648-ssvxf\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.216734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.216688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:41.299387 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:41.299314 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:11:42.353061 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:42.353033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nmbs2" Jul 10 13:11:45.019646 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.019602 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:11:45.150806 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:11:45.150768 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e622f15_e8fb_4fa6_aae3_fb27b934b7cd.slice/crio-a73d11eb8d86d649a8fc553b53a3c9b9ffffc15de95f17f476ea60016416e3da WatchSource:0}: Error finding container a73d11eb8d86d649a8fc553b53a3c9b9ffffc15de95f17f476ea60016416e3da: Status 404 returned error can't find the container with id a73d11eb8d86d649a8fc553b53a3c9b9ffffc15de95f17f476ea60016416e3da Jul 10 13:11:45.582683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.582575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tlxnd" event={"ID":"228f3243-96f6-49a8-935a-6a7bf9d04c84","Type":"ContainerStarted","Data":"f6b3f68c19168b2e3ce5c63045d55805960f07bd9bb29aba0ed62b59f8c3e508"} Jul 10 13:11:45.582908 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.582772 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:45.584723 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.584693 2573 patch_prober.go:28] interesting pod/downloads-68c9cbb497-tlxnd container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Jul 10 13:11:45.584860 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.584745 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-tlxnd" podUID="228f3243-96f6-49a8-935a-6a7bf9d04c84" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" Jul 10 13:11:45.585313 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.585284 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-977766648-ssvxf" event={"ID":"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd","Type":"ContainerStarted","Data":"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9"} Jul 10 13:11:45.585395 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.585333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-977766648-ssvxf" event={"ID":"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd","Type":"ContainerStarted","Data":"a73d11eb8d86d649a8fc553b53a3c9b9ffffc15de95f17f476ea60016416e3da"} Jul 10 13:11:45.613131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.613083 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-tlxnd" podStartSLOduration=3.5712489979999997 podStartE2EDuration="28.613067848s" totalImagesPullingTime="25.04181885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:45.60893585 +0000 UTC m=+103.141538149" watchObservedRunningTime="2026-07-10 13:11:45.613067848 +0000 UTC m=+103.145670148" Jul 10 13:11:45.646245 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:45.646186 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-977766648-ssvxf" podStartSLOduration=5.646171759 podStartE2EDuration="5.646171759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:45.643653591 +0000 UTC m=+103.176255880" watchObservedRunningTime="2026-07-10 13:11:45.646171759 +0000 UTC m=+103.178774057" Jul 10 13:11:46.601997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:46.601967 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-tlxnd" Jul 10 13:11:51.217155 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:51.217113 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:51.217612 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:51.217185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:51.222918 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:51.222895 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:51.620814 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:51.620741 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:11:51.676869 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:51.676839 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:11:56.639982 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:56.639947 2573 generic.go:356] "Generic (PLEG): container finished" podID="b4e79848-1a24-43f0-bee6-1590702cfaca" containerID="8fe7ee86ede0c2d4e1013977ca4ab879ab15da124a7347985cddc16ea6198f17" exitCode=0 Jul 10 13:11:56.640364 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:56.640015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" event={"ID":"b4e79848-1a24-43f0-bee6-1590702cfaca","Type":"ContainerDied","Data":"8fe7ee86ede0c2d4e1013977ca4ab879ab15da124a7347985cddc16ea6198f17"} Jul 10 13:11:56.640364 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:56.640359 2573 scope.go:117] "RemoveContainer" containerID="8fe7ee86ede0c2d4e1013977ca4ab879ab15da124a7347985cddc16ea6198f17" Jul 10 13:11:57.647015 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:11:57.646981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-24fqk" event={"ID":"b4e79848-1a24-43f0-bee6-1590702cfaca","Type":"ContainerStarted","Data":"05f2f23f73616089a263f2cd304df2feb4f8a4fea67e41aacf11afcd291a8200"} Jul 10 13:12:00.660460 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:00.660424 2573 generic.go:356] "Generic (PLEG): container finished" podID="412665cf-e9c5-4da8-897f-5b5d82876e51" containerID="bebd57979551325ca578e1d7a7e21b369274ab5d0993add2d8d094f03e134aa0" exitCode=0 Jul 10 13:12:00.660855 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:00.660500 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" event={"ID":"412665cf-e9c5-4da8-897f-5b5d82876e51","Type":"ContainerDied","Data":"bebd57979551325ca578e1d7a7e21b369274ab5d0993add2d8d094f03e134aa0"} Jul 10 13:12:00.660855 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:00.660845 2573 scope.go:117] "RemoveContainer" containerID="bebd57979551325ca578e1d7a7e21b369274ab5d0993add2d8d094f03e134aa0" Jul 10 13:12:01.667200 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:01.667167 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-hrq4g" event={"ID":"412665cf-e9c5-4da8-897f-5b5d82876e51","Type":"ContainerStarted","Data":"13f953c8c1b2a63fc593719cc255fff397e89e5a3cfe788efdb271be24866f5f"} Jul 10 13:12:06.324959 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.324894 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-655875787f-6ht6h" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerName="registry" containerID="cri-o://93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06" gracePeriod=30 Jul 10 13:12:06.601561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.601502 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:12:06.688435 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.688401 2573 generic.go:356] "Generic (PLEG): container finished" podID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerID="93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06" exitCode=0 Jul 10 13:12:06.688593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.688446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655875787f-6ht6h" event={"ID":"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6","Type":"ContainerDied","Data":"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06"} Jul 10 13:12:06.688593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.688452 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655875787f-6ht6h" Jul 10 13:12:06.688593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.688485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655875787f-6ht6h" event={"ID":"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6","Type":"ContainerDied","Data":"61ddabb59f42a2e31eb01b87a9a2960161797ee7b851c1f0335c7ec6201f3801"} Jul 10 13:12:06.688593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.688522 2573 scope.go:117] "RemoveContainer" containerID="93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06" Jul 10 13:12:06.696865 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.696849 2573 scope.go:117] "RemoveContainer" containerID="93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06" Jul 10 13:12:06.697135 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:12:06.697116 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06\": container with ID starting with 93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06 not found: ID does not exist" containerID="93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06" Jul 10 13:12:06.697213 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.697148 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06"} err="failed to get container status \"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06\": rpc error: code = NotFound desc = could not find container \"93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06\": container with ID starting with 93a9a2652878e3a4147fedd6b797900ef139e114f1e897c0f6e16c9a44334a06 not found: ID does not exist" Jul 10 13:12:06.743470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743445 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743494 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743540 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743565 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743767 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743586 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743767 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.743767 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743719 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg6wt\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt\") pod \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\" (UID: \"65e7898d-2f6b-42d5-83a4-eb23f29bc2c6\") " Jul 10 13:12:06.744065 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.743936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:12:06.744476 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.744420 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:12:06.746353 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.746329 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt" (OuterVolumeSpecName: "kube-api-access-cg6wt") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "kube-api-access-cg6wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:12:06.746353 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.746342 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:12:06.746488 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.746449 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:12:06.746488 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.746465 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:12:06.746564 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.746493 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:12:06.754746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.754717 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" (UID: "65e7898d-2f6b-42d5-83a4-eb23f29bc2c6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:12:06.845115 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845089 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845115 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845114 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6wt\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-kube-api-access-cg6wt\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845124 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-registry-certificates\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845135 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-image-registry-private-configuration\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845144 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-trusted-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845152 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-installation-pull-secrets\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845160 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-bound-sa-token\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:06.845238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:06.845168 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6-ca-trust-extracted\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:07.006485 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:07.006456 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:12:07.010650 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:07.010605 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-655875787f-6ht6h"] Jul 10 13:12:08.970119 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:08.970074 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" path="/var/lib/kubelet/pods/65e7898d-2f6b-42d5-83a4-eb23f29bc2c6/volumes" Jul 10 13:12:15.731835 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:15.731798 2573 generic.go:356] "Generic (PLEG): container finished" podID="e2beeec8-8f39-4756-8e53-4403d5c4d19c" containerID="468d73638ffcab94d7ab8edd34af36084eff93b742ee24b3593ed4fb8ba73e00" exitCode=0 Jul 10 13:12:15.732221 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:15.731878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" event={"ID":"e2beeec8-8f39-4756-8e53-4403d5c4d19c","Type":"ContainerDied","Data":"468d73638ffcab94d7ab8edd34af36084eff93b742ee24b3593ed4fb8ba73e00"} Jul 10 13:12:15.732221 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:15.732159 2573 scope.go:117] "RemoveContainer" containerID="468d73638ffcab94d7ab8edd34af36084eff93b742ee24b3593ed4fb8ba73e00" Jul 10 13:12:16.705934 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:16.705880 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67dcffcf6c-6blgh" podUID="a020c31a-61f8-45e7-b698-697e5a9d3c87" containerName="console" containerID="cri-o://62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13" gracePeriod=15 Jul 10 13:12:16.737773 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:16.737745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zmnmf" event={"ID":"e2beeec8-8f39-4756-8e53-4403d5c4d19c","Type":"ContainerStarted","Data":"92a5b393202a8ad9d91fecfa786cf520044bc4c64ffe9e07e44eb2e7d4319887"} Jul 10 13:12:16.954114 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:16.954090 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:12:17.125674 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.125832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125688 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.125832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125736 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.125832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125752 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.125832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125808 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.125970 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.125849 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rpj8\" (UniqueName: \"kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8\") pod \"a020c31a-61f8-45e7-b698-697e5a9d3c87\" (UID: \"a020c31a-61f8-45e7-b698-697e5a9d3c87\") " Jul 10 13:12:17.126187 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.126158 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config" (OuterVolumeSpecName: "console-config") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:12:17.126273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.126177 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca" (OuterVolumeSpecName: "service-ca") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:12:17.126273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.126183 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:12:17.128155 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.128128 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:12:17.128155 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.128130 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8" (OuterVolumeSpecName: "kube-api-access-6rpj8") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "kube-api-access-6rpj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:12:17.128287 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.128209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a020c31a-61f8-45e7-b698-697e5a9d3c87" (UID: "a020c31a-61f8-45e7-b698-697e5a9d3c87"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:12:17.227200 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227162 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-oauth-serving-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.227200 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227197 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.227342 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227212 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-config\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.227342 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227224 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-serving-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.227342 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227235 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpj8\" (UniqueName: \"kubernetes.io/projected/a020c31a-61f8-45e7-b698-697e5a9d3c87-kube-api-access-6rpj8\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.227342 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.227248 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a020c31a-61f8-45e7-b698-697e5a9d3c87-console-oauth-config\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:12:17.743503 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.743466 2573 generic.go:356] "Generic (PLEG): container finished" podID="a020c31a-61f8-45e7-b698-697e5a9d3c87" containerID="62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13" exitCode=0 Jul 10 13:12:17.743887 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.743514 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dcffcf6c-6blgh" Jul 10 13:12:17.743887 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.743557 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dcffcf6c-6blgh" event={"ID":"a020c31a-61f8-45e7-b698-697e5a9d3c87","Type":"ContainerDied","Data":"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13"} Jul 10 13:12:17.743887 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.743607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dcffcf6c-6blgh" event={"ID":"a020c31a-61f8-45e7-b698-697e5a9d3c87","Type":"ContainerDied","Data":"9e7802ed2fd3311a7c94e805b5e3c97880a2ea8f71f0a957d84e4520ace7b6c9"} Jul 10 13:12:17.743887 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.743654 2573 scope.go:117] "RemoveContainer" containerID="62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13" Jul 10 13:12:17.752084 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.752069 2573 scope.go:117] "RemoveContainer" containerID="62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13" Jul 10 13:12:17.752330 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:12:17.752312 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13\": container with ID starting with 62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13 not found: ID does not exist" containerID="62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13" Jul 10 13:12:17.752386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.752340 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13"} err="failed to get container status \"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13\": rpc error: code = NotFound desc = could not find container \"62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13\": container with ID starting with 62b88bf1548fd7e61cbf1941109ee9bc051023e840fcd4886fd2ba7556816e13 not found: ID does not exist" Jul 10 13:12:17.770947 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.770920 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:12:17.782870 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:17.782846 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67dcffcf6c-6blgh"] Jul 10 13:12:18.967861 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:12:18.967825 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a020c31a-61f8-45e7-b698-697e5a9d3c87" path="/var/lib/kubelet/pods/a020c31a-61f8-45e7-b698-697e5a9d3c87/volumes" Jul 10 13:13:16.322997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:16.322966 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:13:41.342968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.342906 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-977766648-ssvxf" podUID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" containerName="console" containerID="cri-o://c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9" gracePeriod=15 Jul 10 13:13:41.586945 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.586919 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:13:41.676642 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676612 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6zjn\" (UniqueName: \"kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.676783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676664 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.676783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.676783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676717 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.676783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676738 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.676783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.676758 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.677196 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.677152 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config" (OuterVolumeSpecName: "console-config") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:13:41.677196 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.677182 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:13:41.677339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.677206 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:13:41.677339 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.677221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:13:41.678781 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.678761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn" (OuterVolumeSpecName: "kube-api-access-m6zjn") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "kube-api-access-m6zjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:13:41.678879 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.678837 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:13:41.777190 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777168 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert\") pod \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\" (UID: \"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd\") " Jul 10 13:13:41.777358 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777341 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.777407 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777367 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6zjn\" (UniqueName: \"kubernetes.io/projected/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-kube-api-access-m6zjn\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.777407 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777384 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-oauth-config\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.777407 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777398 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-oauth-serving-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.777498 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777412 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-config\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.777498 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.777427 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-trusted-ca-bundle\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:41.778914 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.778895 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" (UID: "1e622f15-e8fb-4fa6-aae3-fb27b934b7cd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:13:41.878611 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:41.878577 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd-console-serving-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:13:42.080553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.080473 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" containerID="c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9" exitCode=0 Jul 10 13:13:42.080553 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.080522 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-977766648-ssvxf" Jul 10 13:13:42.080741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.080555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-977766648-ssvxf" event={"ID":"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd","Type":"ContainerDied","Data":"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9"} Jul 10 13:13:42.080741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.080604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-977766648-ssvxf" event={"ID":"1e622f15-e8fb-4fa6-aae3-fb27b934b7cd","Type":"ContainerDied","Data":"a73d11eb8d86d649a8fc553b53a3c9b9ffffc15de95f17f476ea60016416e3da"} Jul 10 13:13:42.080741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.080624 2573 scope.go:117] "RemoveContainer" containerID="c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9" Jul 10 13:13:42.089593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.089579 2573 scope.go:117] "RemoveContainer" containerID="c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9" Jul 10 13:13:42.089855 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:13:42.089839 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9\": container with ID starting with c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9 not found: ID does not exist" containerID="c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9" Jul 10 13:13:42.089909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.089863 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9"} err="failed to get container status \"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9\": rpc error: code = NotFound desc = could not find container \"c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9\": container with ID starting with c29fc0a7ff11e78ec89ad458d1a81f345752676a0919707610f2f311fd3b04d9 not found: ID does not exist" Jul 10 13:13:42.107593 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.107569 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:13:42.119201 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.119179 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-977766648-ssvxf"] Jul 10 13:13:42.969317 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:13:42.969282 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" path="/var/lib/kubelet/pods/1e622f15-e8fb-4fa6-aae3-fb27b934b7cd/volumes" Jul 10 13:15:02.863201 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:02.863173 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:15:02.869175 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:02.869155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:15:02.871893 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:02.871875 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 13:15:24.224015 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.223981 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh"] Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224294 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" containerName="console" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224304 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" containerName="console" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224320 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a020c31a-61f8-45e7-b698-697e5a9d3c87" containerName="console" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224326 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a020c31a-61f8-45e7-b698-697e5a9d3c87" containerName="console" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224334 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerName="registry" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224340 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerName="registry" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224393 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a020c31a-61f8-45e7-b698-697e5a9d3c87" containerName="console" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224406 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e7898d-2f6b-42d5-83a4-eb23f29bc2c6" containerName="registry" Jul 10 13:15:24.225437 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.224411 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e622f15-e8fb-4fa6-aae3-fb27b934b7cd" containerName="console" Jul 10 13:15:24.226366 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.226350 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.231704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.231672 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 13:15:24.231838 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.231717 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 13:15:24.232131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.232110 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 13:15:24.232249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.232198 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tq2hx\"" Jul 10 13:15:24.232249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.232211 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 13:15:24.251401 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.251381 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh"] Jul 10 13:15:24.315528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.315499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.315701 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.315536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.315701 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.315563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.315701 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.315623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.315701 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.315694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwgn\" (UniqueName: \"kubernetes.io/projected/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-kube-api-access-xgwgn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416228 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416228 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416486 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416486 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwgn\" (UniqueName: \"kubernetes.io/projected/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-kube-api-access-xgwgn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416486 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.416786 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.416765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.418607 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.418586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.418766 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.418750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.418808 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.418784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.426235 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.426209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwgn\" (UniqueName: \"kubernetes.io/projected/292b4d7e-3f17-431a-bfeb-eb7de0e0b920-kube-api-access-xgwgn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-j79kh\" (UID: \"292b4d7e-3f17-431a-bfeb-eb7de0e0b920\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.536675 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.536594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:24.677216 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.677187 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh"] Jul 10 13:15:24.710568 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:24.710539 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b4d7e_3f17_431a_bfeb_eb7de0e0b920.slice/crio-ba555c0f5b5fd7683d10bdaed569c5dff2a38414ab943146a2f3bbd5f727eebf WatchSource:0}: Error finding container ba555c0f5b5fd7683d10bdaed569c5dff2a38414ab943146a2f3bbd5f727eebf: Status 404 returned error can't find the container with id ba555c0f5b5fd7683d10bdaed569c5dff2a38414ab943146a2f3bbd5f727eebf Jul 10 13:15:24.713007 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:24.712988 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:15:25.476680 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:25.476619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" event={"ID":"292b4d7e-3f17-431a-bfeb-eb7de0e0b920","Type":"ContainerStarted","Data":"ba555c0f5b5fd7683d10bdaed569c5dff2a38414ab943146a2f3bbd5f727eebf"} Jul 10 13:15:28.491454 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:28.491407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" event={"ID":"292b4d7e-3f17-431a-bfeb-eb7de0e0b920","Type":"ContainerStarted","Data":"faa6d3104841c5c1c231e9a4adec4a6ee0183e818ecf30c2c44dad2c64dfad39"} Jul 10 13:15:29.497420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:29.497386 2573 generic.go:356] "Generic (PLEG): container finished" podID="292b4d7e-3f17-431a-bfeb-eb7de0e0b920" containerID="faa6d3104841c5c1c231e9a4adec4a6ee0183e818ecf30c2c44dad2c64dfad39" exitCode=0 Jul 10 13:15:29.497890 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:29.497473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" event={"ID":"292b4d7e-3f17-431a-bfeb-eb7de0e0b920","Type":"ContainerDied","Data":"faa6d3104841c5c1c231e9a4adec4a6ee0183e818ecf30c2c44dad2c64dfad39"} Jul 10 13:15:30.503915 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:30.503873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" event={"ID":"292b4d7e-3f17-431a-bfeb-eb7de0e0b920","Type":"ContainerStarted","Data":"051f80e3af3d78df457e63068a540f22eb4f63b99a7967d190ab8e4c8599115f"} Jul 10 13:15:30.504343 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:30.503952 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:30.535649 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:30.535576 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" podStartSLOduration=2.214066003 podStartE2EDuration="6.535556392s" totalImagesPullingTime="3.273455411s" totalInitContainerRuntime="1.048034978s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:30.532791529 +0000 UTC m=+328.065393828" watchObservedRunningTime="2026-07-10 13:15:30.535556392 +0000 UTC m=+328.068158691" Jul 10 13:15:41.510301 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:41.510268 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-j79kh" Jul 10 13:15:45.935710 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.935678 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m"] Jul 10 13:15:45.945563 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.945540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:45.953690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.953672 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 13:15:45.969584 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.969552 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m"] Jul 10 13:15:45.975879 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.975855 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 13:15:45.988337 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.988317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b336ce-f12d-4467-a670-470d51d57edb-tls-certs\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:45.988439 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.988363 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dccwf\" (UniqueName: \"kubernetes.io/projected/82b336ce-f12d-4467-a670-470d51d57edb-kube-api-access-dccwf\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:45.988439 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:45.988395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b336ce-f12d-4467-a670-470d51d57edb-tmp\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.089729 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.089692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b336ce-f12d-4467-a670-470d51d57edb-tmp\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.089891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.089748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b336ce-f12d-4467-a670-470d51d57edb-tls-certs\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.089891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.089790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dccwf\" (UniqueName: \"kubernetes.io/projected/82b336ce-f12d-4467-a670-470d51d57edb-kube-api-access-dccwf\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.091941 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.091917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b336ce-f12d-4467-a670-470d51d57edb-tmp\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.092126 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.092106 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b336ce-f12d-4467-a670-470d51d57edb-tls-certs\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.127916 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.127881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dccwf\" (UniqueName: \"kubernetes.io/projected/82b336ce-f12d-4467-a670-470d51d57edb-kube-api-access-dccwf\") pod \"kube-auth-proxy-76f6cf8c47-bpp2m\" (UID: \"82b336ce-f12d-4467-a670-470d51d57edb\") " pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.255148 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.255056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" Jul 10 13:15:46.333545 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.333517 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mczmh"] Jul 10 13:15:46.336708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.336666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.343976 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.343949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 13:15:46.347789 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.347743 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9p46h\"" Jul 10 13:15:46.373165 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.373140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mczmh"] Jul 10 13:15:46.391804 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.391780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.391896 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.391821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8zvg\" (UniqueName: \"kubernetes.io/projected/473585ca-2dd7-41d2-95e0-ef7aabd71893-kube-api-access-t8zvg\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.455158 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.455073 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m"] Jul 10 13:15:46.457318 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:46.457296 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b336ce_f12d_4467_a670_470d51d57edb.slice/crio-3723c7c69119f13c6565bbeb2249ca301baa5bf3ca14f8061ca9bc0f2ad087d6 WatchSource:0}: Error finding container 3723c7c69119f13c6565bbeb2249ca301baa5bf3ca14f8061ca9bc0f2ad087d6: Status 404 returned error can't find the container with id 3723c7c69119f13c6565bbeb2249ca301baa5bf3ca14f8061ca9bc0f2ad087d6 Jul 10 13:15:46.492604 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.492577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8zvg\" (UniqueName: \"kubernetes.io/projected/473585ca-2dd7-41d2-95e0-ef7aabd71893-kube-api-access-t8zvg\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.492723 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.492706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.492838 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:15:46.492824 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 13:15:46.492914 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:15:46.492889 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert podName:473585ca-2dd7-41d2-95e0-ef7aabd71893 nodeName:}" failed. No retries permitted until 2026-07-10 13:15:46.992868747 +0000 UTC m=+344.525471027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert") pod "odh-model-controller-7998c67dc4-mczmh" (UID: "473585ca-2dd7-41d2-95e0-ef7aabd71893") : secret "odh-model-controller-webhook-cert" not found Jul 10 13:15:46.517859 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.517800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8zvg\" (UniqueName: \"kubernetes.io/projected/473585ca-2dd7-41d2-95e0-ef7aabd71893-kube-api-access-t8zvg\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.570141 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.570115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" event={"ID":"82b336ce-f12d-4467-a670-470d51d57edb","Type":"ContainerStarted","Data":"3723c7c69119f13c6565bbeb2249ca301baa5bf3ca14f8061ca9bc0f2ad087d6"} Jul 10 13:15:46.996483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:46.996453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:46.996847 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:15:46.996595 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 13:15:46.996847 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:15:46.996677 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert podName:473585ca-2dd7-41d2-95e0-ef7aabd71893 nodeName:}" failed. No retries permitted until 2026-07-10 13:15:47.996663726 +0000 UTC m=+345.529266002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert") pod "odh-model-controller-7998c67dc4-mczmh" (UID: "473585ca-2dd7-41d2-95e0-ef7aabd71893") : secret "odh-model-controller-webhook-cert" not found Jul 10 13:15:48.004569 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.004537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:48.006928 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.006910 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/473585ca-2dd7-41d2-95e0-ef7aabd71893-cert\") pod \"odh-model-controller-7998c67dc4-mczmh\" (UID: \"473585ca-2dd7-41d2-95e0-ef7aabd71893\") " pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:48.148349 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.148314 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:48.194204 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.193874 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5vrhw"] Jul 10 13:15:48.196528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.196507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.219690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.218988 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 13:15:48.258671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.258588 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5vrhw"] Jul 10 13:15:48.307929 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.307901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-tls-certs\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.308075 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.307952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5ph\" (UniqueName: \"kubernetes.io/projected/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-kube-api-access-jd5ph\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.351282 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.351240 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mczmh"] Jul 10 13:15:48.357076 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:48.357046 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473585ca_2dd7_41d2_95e0_ef7aabd71893.slice/crio-e715182d9c379aa7217ff6610491df1c0d2c5ecb26d9580c23ca929e297e2774 WatchSource:0}: Error finding container e715182d9c379aa7217ff6610491df1c0d2c5ecb26d9580c23ca929e297e2774: Status 404 returned error can't find the container with id e715182d9c379aa7217ff6610491df1c0d2c5ecb26d9580c23ca929e297e2774 Jul 10 13:15:48.409362 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.409328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-tls-certs\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.409522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.409432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5ph\" (UniqueName: \"kubernetes.io/projected/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-kube-api-access-jd5ph\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.411980 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.411957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-tls-certs\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.433445 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.433408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5ph\" (UniqueName: \"kubernetes.io/projected/2302ac8f-d09d-47bd-bd9b-be9aa5c943ae-kube-api-access-jd5ph\") pod \"model-serving-api-54d89bd5fc-5vrhw\" (UID: \"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae\") " pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.512231 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.512144 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:48.580369 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.580337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" event={"ID":"473585ca-2dd7-41d2-95e0-ef7aabd71893","Type":"ContainerStarted","Data":"e715182d9c379aa7217ff6610491df1c0d2c5ecb26d9580c23ca929e297e2774"} Jul 10 13:15:48.655443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:48.655406 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5vrhw"] Jul 10 13:15:48.658014 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:48.657983 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2302ac8f_d09d_47bd_bd9b_be9aa5c943ae.slice/crio-b5ab573aab8f0f0e9ff0c74d4de930b2a55aa0cf77b35da9213c1515de4a7864 WatchSource:0}: Error finding container b5ab573aab8f0f0e9ff0c74d4de930b2a55aa0cf77b35da9213c1515de4a7864: Status 404 returned error can't find the container with id b5ab573aab8f0f0e9ff0c74d4de930b2a55aa0cf77b35da9213c1515de4a7864 Jul 10 13:15:49.406571 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.406529 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:15:49.409880 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.409858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.413880 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.413852 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 13:15:49.414438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.414408 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8tq4d\"" Jul 10 13:15:49.414438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.414429 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 13:15:49.414438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.414437 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 13:15:49.414690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.414429 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 13:15:49.434983 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.434956 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.517868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttt6s\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.517938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.517985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.518042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.518070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.518126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.518248 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.518150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.589926 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.589880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" event={"ID":"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae","Type":"ContainerStarted","Data":"b5ab573aab8f0f0e9ff0c74d4de930b2a55aa0cf77b35da9213c1515de4a7864"} Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttt6s\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.620581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.619800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.621145 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.620928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.624320 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.624274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.624970 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.624927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.625495 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.625450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.627086 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.627044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.630379 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.630345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.647064 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.646999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttt6s\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s\") pod \"istiod-openshift-gateway-7574459656-jb2gc\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:49.721751 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:49.721453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:50.330549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:50.330519 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:15:50.596963 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:50.596869 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" event={"ID":"82b336ce-f12d-4467-a670-470d51d57edb","Type":"ContainerStarted","Data":"733d7e18cb61102f7be0edd6a5cb90e400ee7b6ea8c912b6cf0bd4cd35b415d4"} Jul 10 13:15:50.653295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:50.653239 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76f6cf8c47-bpp2m" podStartSLOduration=1.9287026809999999 podStartE2EDuration="5.653222563s" totalImagesPullingTime="3.724519882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:50.652601371 +0000 UTC m=+348.185203671" watchObservedRunningTime="2026-07-10 13:15:50.653222563 +0000 UTC m=+348.185824913" Jul 10 13:15:50.949159 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:50.949118 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccf8825_0d4d_4d17_94d0_3d0bccc44ad7.slice/crio-221be1d57b53ef5ee966bbbaa489a96697481b7c21621567771c57a6e9e28668 WatchSource:0}: Error finding container 221be1d57b53ef5ee966bbbaa489a96697481b7c21621567771c57a6e9e28668: Status 404 returned error can't find the container with id 221be1d57b53ef5ee966bbbaa489a96697481b7c21621567771c57a6e9e28668 Jul 10 13:15:51.604897 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:51.604855 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" event={"ID":"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7","Type":"ContainerStarted","Data":"221be1d57b53ef5ee966bbbaa489a96697481b7c21621567771c57a6e9e28668"} Jul 10 13:15:52.612382 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.612300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" event={"ID":"473585ca-2dd7-41d2-95e0-ef7aabd71893","Type":"ContainerStarted","Data":"3fc8fd4f46cef3ed8428c826ad142b41e83ff02c0120eef997a93fa2aa4529b9"} Jul 10 13:15:52.612896 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.612418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:15:52.614304 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.614243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" event={"ID":"2302ac8f-d09d-47bd-bd9b-be9aa5c943ae","Type":"ContainerStarted","Data":"ce9db441378b998d8ac691fa67a34a4d887f00d4df2f00610e80ed65aa37b559"} Jul 10 13:15:52.614465 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.614400 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:15:52.662448 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.662383 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" podStartSLOduration=3.149404019 podStartE2EDuration="6.66236203s" totalImagesPullingTime="3.512958011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:52.661404417 +0000 UTC m=+350.194006717" watchObservedRunningTime="2026-07-10 13:15:52.66236203 +0000 UTC m=+350.194964330" Jul 10 13:15:52.724406 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:52.724343 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" podStartSLOduration=1.5222670649999999 podStartE2EDuration="4.724321032s" totalImagesPullingTime="3.202053967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:52.721353375 +0000 UTC m=+350.253955673" watchObservedRunningTime="2026-07-10 13:15:52.724321032 +0000 UTC m=+350.256923330" Jul 10 13:15:53.697918 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:53.697875 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:15:53.698274 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:53.697962 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:15:54.624699 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:54.624658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" event={"ID":"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7","Type":"ContainerStarted","Data":"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563"} Jul 10 13:15:54.624909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:54.624853 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:54.626533 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:54.626505 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jb2gc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 13:15:54.626655 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:54.626561 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:15:54.658110 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:54.658056 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" podStartSLOduration=2.9122215970000003 podStartE2EDuration="5.658024162s" totalImagesPullingTime="2.745802565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:54.656174642 +0000 UTC m=+352.188776942" watchObservedRunningTime="2026-07-10 13:15:54.658024162 +0000 UTC m=+352.190626452" Jul 10 13:15:55.630282 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:55.630239 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jb2gc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 13:15:55.630687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:55.630296 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:15:58.389571 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.389532 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn"] Jul 10 13:15:58.393058 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.393033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.410823 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:15:58.410788 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-wmdwn\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="e5089a22-2976-408a-9a13-08390cd73ee2" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.410913 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.410824 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 13:15:58.410913 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.410864 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9wnz8\"" Jul 10 13:15:58.449068 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.449048 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn"] Jul 10 13:15:58.502577 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.502553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5089a22-2976-408a-9a13-08390cd73ee2-cert\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.502705 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.502591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtt4\" (UniqueName: \"kubernetes.io/projected/e5089a22-2976-408a-9a13-08390cd73ee2-kube-api-access-hdtt4\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.603456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.603418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5089a22-2976-408a-9a13-08390cd73ee2-cert\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.603456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.603462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtt4\" (UniqueName: \"kubernetes.io/projected/e5089a22-2976-408a-9a13-08390cd73ee2-kube-api-access-hdtt4\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.605917 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.605894 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5089a22-2976-408a-9a13-08390cd73ee2-cert\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.631019 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.630994 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:15:58.633585 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.633556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtt4\" (UniqueName: \"kubernetes.io/projected/e5089a22-2976-408a-9a13-08390cd73ee2-kube-api-access-hdtt4\") pod \"kserve-controller-manager-d6f4b65b8-wmdwn\" (UID: \"e5089a22-2976-408a-9a13-08390cd73ee2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.703392 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.703324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:15:58.869941 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:58.869906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn"] Jul 10 13:15:58.873907 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:15:58.873879 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5089a22_2976_408a_9a13_08390cd73ee2.slice/crio-dfd38176507bab70f9dd689546854b85519e9143b9c5a9fb22c5cdaab1f133f6 WatchSource:0}: Error finding container dfd38176507bab70f9dd689546854b85519e9143b9c5a9fb22c5cdaab1f133f6: Status 404 returned error can't find the container with id dfd38176507bab70f9dd689546854b85519e9143b9c5a9fb22c5cdaab1f133f6 Jul 10 13:15:59.648739 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:15:59.648702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" event={"ID":"e5089a22-2976-408a-9a13-08390cd73ee2","Type":"ContainerStarted","Data":"dfd38176507bab70f9dd689546854b85519e9143b9c5a9fb22c5cdaab1f133f6"} Jul 10 13:16:01.580391 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:01.580357 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:16:01.580766 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:01.580546 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" containerID="cri-o://d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563" gracePeriod=30 Jul 10 13:16:01.630434 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:01.630409 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jb2gc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" start-of-body= Jul 10 13:16:01.630564 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:01.630456 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Jul 10 13:16:02.274154 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.274129 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:16:02.336951 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.336925 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.336961 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.336993 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.337016 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.337087 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.337204 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttt6s\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.337256 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap\") pod \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\" (UID: \"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7\") " Jul 10 13:16:02.337924 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.337859 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:16:02.339564 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.339528 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s" (OuterVolumeSpecName: "kube-api-access-ttt6s") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "kube-api-access-ttt6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:16:02.339817 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.339784 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:16:02.339817 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.339790 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs" (OuterVolumeSpecName: "local-certs") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:16:02.339817 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.339811 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:16:02.339983 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.339868 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts" (OuterVolumeSpecName: "cacerts") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:16:02.340131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.340116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" (UID: "5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:16:02.438816 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438749 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttt6s\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-kube-api-access-ttt6s\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.438816 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438783 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-ca-configmap\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.438816 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438798 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-token\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.438816 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438813 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-kubeconfig\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.439021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438826 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-local-certs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.439021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438838 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-cacerts\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.439021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.438850 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7-istio-csr-dns-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:02.664718 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.664685 2573 generic.go:356] "Generic (PLEG): container finished" podID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerID="d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563" exitCode=0 Jul 10 13:16:02.665152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.664732 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" Jul 10 13:16:02.665152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.664776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" event={"ID":"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7","Type":"ContainerDied","Data":"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563"} Jul 10 13:16:02.665152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.664844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc" event={"ID":"5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7","Type":"ContainerDied","Data":"221be1d57b53ef5ee966bbbaa489a96697481b7c21621567771c57a6e9e28668"} Jul 10 13:16:02.665152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.664878 2573 scope.go:117] "RemoveContainer" containerID="d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563" Jul 10 13:16:02.666492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.666468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" event={"ID":"e5089a22-2976-408a-9a13-08390cd73ee2","Type":"ContainerStarted","Data":"5b67ffd7e07ab6ad423500898d0cd396fea0ddf6926b7aa04f4d0e44404e2d86"} Jul 10 13:16:02.666588 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.666523 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:16:02.673373 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.673357 2573 scope.go:117] "RemoveContainer" containerID="d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563" Jul 10 13:16:02.673606 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:16:02.673588 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563\": container with ID starting with d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563 not found: ID does not exist" containerID="d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563" Jul 10 13:16:02.673704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.673614 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563"} err="failed to get container status \"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563\": rpc error: code = NotFound desc = could not find container \"d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563\": container with ID starting with d880ac9aa2969d4e1f2d898a45cefc8bb6ada134fff861376c7dbb7ed96d6563 not found: ID does not exist" Jul 10 13:16:02.720098 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.720020 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:16:02.729778 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.729757 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jb2gc"] Jul 10 13:16:02.765426 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.765374 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" podStartSLOduration=1.452353496 podStartE2EDuration="4.765357886s" totalImagesPullingTime="3.31300439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:16:02.765000001 +0000 UTC m=+360.297602299" watchObservedRunningTime="2026-07-10 13:16:02.765357886 +0000 UTC m=+360.297960186" Jul 10 13:16:02.967966 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:02.967931 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" path="/var/lib/kubelet/pods/5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7/volumes" Jul 10 13:16:03.621258 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:03.621229 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mczmh" Jul 10 13:16:03.623073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:03.623054 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5vrhw" Jul 10 13:16:33.675974 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:33.675945 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wmdwn" Jul 10 13:16:47.193920 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.193886 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:47.194459 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.194389 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" Jul 10 13:16:47.194459 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.194409 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" Jul 10 13:16:47.194572 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.194509 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccf8825-0d4d-4d17-94d0-3d0bccc44ad7" containerName="discovery" Jul 10 13:16:47.196683 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.196663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:47.201590 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.201565 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wrj2v\"" Jul 10 13:16:47.201746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.201593 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 13:16:47.201746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.201593 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 13:16:47.208052 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.208032 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:47.288414 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.288391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvswr\" (UniqueName: \"kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr\") pod \"kuadrant-operator-catalog-kmwtg\" (UID: \"7ffd7144-8ce9-4289-b3b2-b44e2d20350a\") " pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:47.389614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.389579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvswr\" (UniqueName: \"kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr\") pod \"kuadrant-operator-catalog-kmwtg\" (UID: \"7ffd7144-8ce9-4289-b3b2-b44e2d20350a\") " pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:47.407028 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.406999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvswr\" (UniqueName: \"kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr\") pod \"kuadrant-operator-catalog-kmwtg\" (UID: \"7ffd7144-8ce9-4289-b3b2-b44e2d20350a\") " pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:47.507775 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.507699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:47.638354 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.638329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:47.640959 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:16:47.640926 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffd7144_8ce9_4289_b3b2_b44e2d20350a.slice/crio-2a0e0611edd548ca84a8392dcff3a8148cf758ae7a843ed128f7ed0944205714 WatchSource:0}: Error finding container 2a0e0611edd548ca84a8392dcff3a8148cf758ae7a843ed128f7ed0944205714: Status 404 returned error can't find the container with id 2a0e0611edd548ca84a8392dcff3a8148cf758ae7a843ed128f7ed0944205714 Jul 10 13:16:47.667609 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.667588 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:47.750766 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.750740 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxd5n"] Jul 10 13:16:47.753698 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.753682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:47.762009 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.761981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxd5n"] Jul 10 13:16:47.794219 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.794194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhk4\" (UniqueName: \"kubernetes.io/projected/c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4-kube-api-access-mfhk4\") pod \"kuadrant-operator-catalog-wxd5n\" (UID: \"c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4\") " pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:47.859854 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.859822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" event={"ID":"7ffd7144-8ce9-4289-b3b2-b44e2d20350a","Type":"ContainerStarted","Data":"2a0e0611edd548ca84a8392dcff3a8148cf758ae7a843ed128f7ed0944205714"} Jul 10 13:16:47.895409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.895384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhk4\" (UniqueName: \"kubernetes.io/projected/c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4-kube-api-access-mfhk4\") pod \"kuadrant-operator-catalog-wxd5n\" (UID: \"c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4\") " pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:47.903476 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:47.903450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhk4\" (UniqueName: \"kubernetes.io/projected/c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4-kube-api-access-mfhk4\") pod \"kuadrant-operator-catalog-wxd5n\" (UID: \"c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4\") " pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:48.064399 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:48.064326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:48.199448 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:48.199424 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxd5n"] Jul 10 13:16:48.202070 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:16:48.202046 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16cea15_9dfd_4e0e_87d0_bfce4ed2e0d4.slice/crio-37120af5996394c6dd8e60599954aee45e107a976154e039ebde5da2e9c8812a WatchSource:0}: Error finding container 37120af5996394c6dd8e60599954aee45e107a976154e039ebde5da2e9c8812a: Status 404 returned error can't find the container with id 37120af5996394c6dd8e60599954aee45e107a976154e039ebde5da2e9c8812a Jul 10 13:16:48.866438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:48.866389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" event={"ID":"c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4","Type":"ContainerStarted","Data":"37120af5996394c6dd8e60599954aee45e107a976154e039ebde5da2e9c8812a"} Jul 10 13:16:50.878302 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:50.878264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" event={"ID":"c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4","Type":"ContainerStarted","Data":"64319da17b32ab8845f1bafd976e61f1a03682ab36d1c07ea2ed7b56eee2bcca"} Jul 10 13:16:50.879772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:50.879745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" event={"ID":"7ffd7144-8ce9-4289-b3b2-b44e2d20350a","Type":"ContainerStarted","Data":"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320"} Jul 10 13:16:50.879884 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:50.879780 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" podUID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" containerName="registry-server" containerID="cri-o://7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320" gracePeriod=2 Jul 10 13:16:50.895064 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:50.895024 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" podStartSLOduration=2.083054494 podStartE2EDuration="3.895004755s" totalImagesPullingTime="1.811950261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:16:50.894210782 +0000 UTC m=+408.426813081" watchObservedRunningTime="2026-07-10 13:16:50.895004755 +0000 UTC m=+408.427607054" Jul 10 13:16:50.910586 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:50.910547 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" podStartSLOduration=1.5336170569999998 podStartE2EDuration="3.910533855s" totalImagesPullingTime="2.376916798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:16:50.909498788 +0000 UTC m=+408.442101088" watchObservedRunningTime="2026-07-10 13:16:50.910533855 +0000 UTC m=+408.443136154" Jul 10 13:16:51.123666 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.123620 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:51.224895 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.224828 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvswr\" (UniqueName: \"kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr\") pod \"7ffd7144-8ce9-4289-b3b2-b44e2d20350a\" (UID: \"7ffd7144-8ce9-4289-b3b2-b44e2d20350a\") " Jul 10 13:16:51.226941 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.226910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr" (OuterVolumeSpecName: "kube-api-access-dvswr") pod "7ffd7144-8ce9-4289-b3b2-b44e2d20350a" (UID: "7ffd7144-8ce9-4289-b3b2-b44e2d20350a"). InnerVolumeSpecName "kube-api-access-dvswr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:16:51.325341 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.325312 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvswr\" (UniqueName: \"kubernetes.io/projected/7ffd7144-8ce9-4289-b3b2-b44e2d20350a-kube-api-access-dvswr\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:16:51.885507 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.885473 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" containerID="7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320" exitCode=0 Jul 10 13:16:51.885919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.885520 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" Jul 10 13:16:51.885919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.885561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" event={"ID":"7ffd7144-8ce9-4289-b3b2-b44e2d20350a","Type":"ContainerDied","Data":"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320"} Jul 10 13:16:51.885919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.885617 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kmwtg" event={"ID":"7ffd7144-8ce9-4289-b3b2-b44e2d20350a","Type":"ContainerDied","Data":"2a0e0611edd548ca84a8392dcff3a8148cf758ae7a843ed128f7ed0944205714"} Jul 10 13:16:51.885919 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.885661 2573 scope.go:117] "RemoveContainer" containerID="7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320" Jul 10 13:16:51.894708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.894690 2573 scope.go:117] "RemoveContainer" containerID="7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320" Jul 10 13:16:51.894970 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:16:51.894950 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320\": container with ID starting with 7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320 not found: ID does not exist" containerID="7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320" Jul 10 13:16:51.895030 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.894978 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320"} err="failed to get container status \"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320\": rpc error: code = NotFound desc = could not find container \"7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320\": container with ID starting with 7d33233b4c347eed3976994a99117ecd9ca0df072a4c08dd6b07958cced5c320 not found: ID does not exist" Jul 10 13:16:51.906218 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.906193 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:51.911780 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:51.911761 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kmwtg"] Jul 10 13:16:52.969613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:52.969578 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" path="/var/lib/kubelet/pods/7ffd7144-8ce9-4289-b3b2-b44e2d20350a/volumes" Jul 10 13:16:58.064847 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:58.064812 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:58.065235 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:58.064870 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:58.087227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:58.087201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:16:58.937447 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:16:58.937418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wxd5n" Jul 10 13:17:21.739426 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.739389 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd"] Jul 10 13:17:21.739929 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.739738 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" containerName="registry-server" Jul 10 13:17:21.739929 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.739750 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" containerName="registry-server" Jul 10 13:17:21.739929 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.739823 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffd7144-8ce9-4289-b3b2-b44e2d20350a" containerName="registry-server" Jul 10 13:17:21.744335 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.744317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.747258 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.747235 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmfzh\"" Jul 10 13:17:21.759959 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.759938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd"] Jul 10 13:17:21.878952 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.878922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.879128 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.878981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvt9n\" (UniqueName: \"kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.979658 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.979603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvt9n\" (UniqueName: \"kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.979813 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.979713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.980059 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.980040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:21.990011 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:21.989949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvt9n\" (UniqueName: \"kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:22.054017 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:22.053986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:22.262823 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:22.262798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd"] Jul 10 13:17:22.265802 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:22.265772 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062b79f3_6ad3_4816_969b_3348451f6c31.slice/crio-eeb6372edf6f3a91ea536c27a196ec68fe71dd47935c15b52154eb9c7f69d759 WatchSource:0}: Error finding container eeb6372edf6f3a91ea536c27a196ec68fe71dd47935c15b52154eb9c7f69d759: Status 404 returned error can't find the container with id eeb6372edf6f3a91ea536c27a196ec68fe71dd47935c15b52154eb9c7f69d759 Jul 10 13:17:23.012802 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:23.012769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" event={"ID":"062b79f3-6ad3-4816-969b-3348451f6c31","Type":"ContainerStarted","Data":"eeb6372edf6f3a91ea536c27a196ec68fe71dd47935c15b52154eb9c7f69d759"} Jul 10 13:17:24.817730 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:24.817649 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dmbln"] Jul 10 13:17:24.823877 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:24.823862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:24.836304 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:24.836151 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s4lht\"" Jul 10 13:17:24.837880 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:24.837858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dmbln"] Jul 10 13:17:25.003367 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:25.003332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486nv\" (UniqueName: \"kubernetes.io/projected/a93255e3-73ff-4aca-bf48-7aca5665f256-kube-api-access-486nv\") pod \"authorino-operator-75cc8d5cb7-dmbln\" (UID: \"a93255e3-73ff-4aca-bf48-7aca5665f256\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:25.104573 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:25.104480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486nv\" (UniqueName: \"kubernetes.io/projected/a93255e3-73ff-4aca-bf48-7aca5665f256-kube-api-access-486nv\") pod \"authorino-operator-75cc8d5cb7-dmbln\" (UID: \"a93255e3-73ff-4aca-bf48-7aca5665f256\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:25.124956 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:25.124921 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-486nv\" (UniqueName: \"kubernetes.io/projected/a93255e3-73ff-4aca-bf48-7aca5665f256-kube-api-access-486nv\") pod \"authorino-operator-75cc8d5cb7-dmbln\" (UID: \"a93255e3-73ff-4aca-bf48-7aca5665f256\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:25.133702 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:25.133678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:25.319256 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:25.319228 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dmbln"] Jul 10 13:17:25.322142 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:25.322114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93255e3_73ff_4aca_bf48_7aca5665f256.slice/crio-519ece714112d49583a0a3a92debf92861048e632c74f19401e57aee60d25aff WatchSource:0}: Error finding container 519ece714112d49583a0a3a92debf92861048e632c74f19401e57aee60d25aff: Status 404 returned error can't find the container with id 519ece714112d49583a0a3a92debf92861048e632c74f19401e57aee60d25aff Jul 10 13:17:26.026822 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.026786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" event={"ID":"a93255e3-73ff-4aca-bf48-7aca5665f256","Type":"ContainerStarted","Data":"519ece714112d49583a0a3a92debf92861048e632c74f19401e57aee60d25aff"} Jul 10 13:17:26.667348 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.667273 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss"] Jul 10 13:17:26.671216 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.671191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:26.674424 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.674401 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 13:17:26.674808 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.674781 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pwvvt\"" Jul 10 13:17:26.681926 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.681579 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss"] Jul 10 13:17:26.819955 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.819783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m46b\" (UniqueName: \"kubernetes.io/projected/91116b99-7e90-421e-97af-49ae06aa6b29-kube-api-access-8m46b\") pod \"dns-operator-controller-manager-864dd78c4c-92bss\" (UID: \"91116b99-7e90-421e-97af-49ae06aa6b29\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:26.921216 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.921129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m46b\" (UniqueName: \"kubernetes.io/projected/91116b99-7e90-421e-97af-49ae06aa6b29-kube-api-access-8m46b\") pod \"dns-operator-controller-manager-864dd78c4c-92bss\" (UID: \"91116b99-7e90-421e-97af-49ae06aa6b29\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:26.930821 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.930791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m46b\" (UniqueName: \"kubernetes.io/projected/91116b99-7e90-421e-97af-49ae06aa6b29-kube-api-access-8m46b\") pod \"dns-operator-controller-manager-864dd78c4c-92bss\" (UID: \"91116b99-7e90-421e-97af-49ae06aa6b29\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:26.989130 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:26.989089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:29.045910 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.045473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" event={"ID":"a93255e3-73ff-4aca-bf48-7aca5665f256","Type":"ContainerStarted","Data":"e06b0caca744e8e2b56ae179f28d4e39cb4a80d727ad7ccb14aaae4f773a8d84"} Jul 10 13:17:29.045910 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.045572 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:29.047772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.047704 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" event={"ID":"062b79f3-6ad3-4816-969b-3348451f6c31","Type":"ContainerStarted","Data":"2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a"} Jul 10 13:17:29.047772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.047756 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:29.068191 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.068107 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" podStartSLOduration=1.440698654 podStartE2EDuration="5.068086714s" totalImagesPullingTime="3.62738806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:29.065844989 +0000 UTC m=+446.598447288" watchObservedRunningTime="2026-07-10 13:17:29.068086714 +0000 UTC m=+446.600689014" Jul 10 13:17:29.089908 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.089869 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" podStartSLOduration=1.393219043 podStartE2EDuration="8.089855849s" totalImagesPullingTime="6.696636806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:29.089255089 +0000 UTC m=+446.621857389" watchObservedRunningTime="2026-07-10 13:17:29.089855849 +0000 UTC m=+446.622458147" Jul 10 13:17:29.109658 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.109620 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss"] Jul 10 13:17:29.112382 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:29.112362 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91116b99_7e90_421e_97af_49ae06aa6b29.slice/crio-0a41f9332afc7245a8bcaa831823fdbe1c7b9d53c83daab90261e55db0d61e89 WatchSource:0}: Error finding container 0a41f9332afc7245a8bcaa831823fdbe1c7b9d53c83daab90261e55db0d61e89: Status 404 returned error can't find the container with id 0a41f9332afc7245a8bcaa831823fdbe1c7b9d53c83daab90261e55db0d61e89 Jul 10 13:17:29.325972 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.325901 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5"] Jul 10 13:17:29.329821 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.329796 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:29.332388 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.332367 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-q4xtg\"" Jul 10 13:17:29.340996 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.340971 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5"] Jul 10 13:17:29.442409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.442377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgvq\" (UniqueName: \"kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" (UID: \"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:29.543087 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.543031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgvq\" (UniqueName: \"kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" (UID: \"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:29.552176 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.552148 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgvq\" (UniqueName: \"kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq\") pod \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" (UID: \"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:29.641743 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.641714 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:29.778255 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:29.778226 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5"] Jul 10 13:17:29.781599 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:29.781572 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca937c9_eeb8_4fc1_8cb0_add95b6529d5.slice/crio-58504c5814157271498e6221ddd471809ceb907216fc00578cc8bad27724fbfb WatchSource:0}: Error finding container 58504c5814157271498e6221ddd471809ceb907216fc00578cc8bad27724fbfb: Status 404 returned error can't find the container with id 58504c5814157271498e6221ddd471809ceb907216fc00578cc8bad27724fbfb Jul 10 13:17:30.053611 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:30.053526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" event={"ID":"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5","Type":"ContainerStarted","Data":"58504c5814157271498e6221ddd471809ceb907216fc00578cc8bad27724fbfb"} Jul 10 13:17:30.054811 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:30.054783 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" event={"ID":"91116b99-7e90-421e-97af-49ae06aa6b29","Type":"ContainerStarted","Data":"0a41f9332afc7245a8bcaa831823fdbe1c7b9d53c83daab90261e55db0d61e89"} Jul 10 13:17:31.912220 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.912177 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss"] Jul 10 13:17:31.916410 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.916384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:31.919682 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.919520 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fd6r5\"" Jul 10 13:17:31.919682 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.919552 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 13:17:31.920199 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.920086 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 13:17:31.922608 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:31.922586 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss"] Jul 10 13:17:32.063678 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.063641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfb2\" (UniqueName: \"kubernetes.io/projected/d42c7116-492f-4386-b009-388cf862866c-kube-api-access-jrfb2\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.063866 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.063818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d42c7116-492f-4386-b009-388cf862866c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.063866 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.063849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42c7116-492f-4386-b009-388cf862866c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.164760 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.164672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfb2\" (UniqueName: \"kubernetes.io/projected/d42c7116-492f-4386-b009-388cf862866c-kube-api-access-jrfb2\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.164903 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.164769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d42c7116-492f-4386-b009-388cf862866c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.164903 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.164800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42c7116-492f-4386-b009-388cf862866c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.165569 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.165545 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d42c7116-492f-4386-b009-388cf862866c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.167478 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.167449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42c7116-492f-4386-b009-388cf862866c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.174408 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.174384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfb2\" (UniqueName: \"kubernetes.io/projected/d42c7116-492f-4386-b009-388cf862866c-kube-api-access-jrfb2\") pod \"kuadrant-console-plugin-66b5774985-qvtss\" (UID: \"d42c7116-492f-4386-b009-388cf862866c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.230383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.230337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" Jul 10 13:17:32.433371 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:32.433344 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss"] Jul 10 13:17:32.436030 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:32.436005 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd42c7116_492f_4386_b009_388cf862866c.slice/crio-77f6e25e6dd7bd42eda01c17969e8fbcea091f6a8bed2f2a1698ef8ec4fafefd WatchSource:0}: Error finding container 77f6e25e6dd7bd42eda01c17969e8fbcea091f6a8bed2f2a1698ef8ec4fafefd: Status 404 returned error can't find the container with id 77f6e25e6dd7bd42eda01c17969e8fbcea091f6a8bed2f2a1698ef8ec4fafefd Jul 10 13:17:33.070137 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.070098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" event={"ID":"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5","Type":"ContainerStarted","Data":"5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d"} Jul 10 13:17:33.070596 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.070163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:33.071649 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.071607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" event={"ID":"91116b99-7e90-421e-97af-49ae06aa6b29","Type":"ContainerStarted","Data":"e4297dc1aae0a3c624ddbaa6c809769131b18ae3bb095ee1d6e4e0a21acaf1e8"} Jul 10 13:17:33.071778 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.071706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:33.072915 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.072888 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" event={"ID":"d42c7116-492f-4386-b009-388cf862866c","Type":"ContainerStarted","Data":"77f6e25e6dd7bd42eda01c17969e8fbcea091f6a8bed2f2a1698ef8ec4fafefd"} Jul 10 13:17:33.086977 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.086923 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" podStartSLOduration=1.569623398 podStartE2EDuration="4.086906926s" totalImagesPullingTime="2.517283528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:33.086734013 +0000 UTC m=+450.619336314" watchObservedRunningTime="2026-07-10 13:17:33.086906926 +0000 UTC m=+450.619509226" Jul 10 13:17:33.106220 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:33.106181 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" podStartSLOduration=3.91631065 podStartE2EDuration="7.106170254s" totalImagesPullingTime="3.189859604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:33.104751711 +0000 UTC m=+450.637354010" watchObservedRunningTime="2026-07-10 13:17:33.106170254 +0000 UTC m=+450.638772614" Jul 10 13:17:40.057113 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:40.057078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:40.057575 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:40.057165 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dmbln" Jul 10 13:17:42.069946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.069916 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd"] Jul 10 13:17:42.070370 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.070117 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" containerName="manager" containerID="cri-o://2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a" gracePeriod=2 Jul 10 13:17:42.101003 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.100974 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd"] Jul 10 13:17:42.127854 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.126903 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5"] Jul 10 13:17:42.127854 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.127100 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" containerName="manager" containerID="cri-o://5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d" gracePeriod=2 Jul 10 13:17:42.129178 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.129079 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:42.132606 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.132419 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"062b79f3-6ad3-4816-969b-3348451f6c31\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wmczd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:42.142718 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.139736 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:42.142718 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.142048 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5"] Jul 10 13:17:42.186094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186060 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:17:42.186581 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186565 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" containerName="manager" Jul 10 13:17:42.186672 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186584 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" containerName="manager" Jul 10 13:17:42.186672 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186615 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" containerName="manager" Jul 10 13:17:42.186672 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186624 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" containerName="manager" Jul 10 13:17:42.186834 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186728 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" containerName="manager" Jul 10 13:17:42.186834 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.186742 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" containerName="manager" Jul 10 13:17:42.191297 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.191278 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.198493 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.198466 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:42.203808 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.203788 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r"] Jul 10 13:17:42.208968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.208813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:42.211134 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.211113 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:17:42.230087 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.230026 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r"] Jul 10 13:17:42.233851 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.233813 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:42.236197 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.236171 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:42.249277 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.249178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.249277 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.249255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwrz\" (UniqueName: \"kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.249424 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.249289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmm8\" (UniqueName: \"kubernetes.io/projected/e549be6b-ab7b-4a33-82f2-9bcfb91272ef-kube-api-access-qkmm8\") pod \"limitador-operator-controller-manager-859d7b78bd-l5t8r\" (UID: \"e549be6b-ab7b-4a33-82f2-9bcfb91272ef\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:42.280037 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.279478 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:42.349940 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.349861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwrz\" (UniqueName: \"kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.349940 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.349911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmm8\" (UniqueName: \"kubernetes.io/projected/e549be6b-ab7b-4a33-82f2-9bcfb91272ef-kube-api-access-qkmm8\") pod \"limitador-operator-controller-manager-859d7b78bd-l5t8r\" (UID: \"e549be6b-ab7b-4a33-82f2-9bcfb91272ef\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:42.350142 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.350001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.350355 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.350335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.367981 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.367945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmm8\" (UniqueName: \"kubernetes.io/projected/e549be6b-ab7b-4a33-82f2-9bcfb91272ef-kube-api-access-qkmm8\") pod \"limitador-operator-controller-manager-859d7b78bd-l5t8r\" (UID: \"e549be6b-ab7b-4a33-82f2-9bcfb91272ef\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:42.367981 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.367945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwrz\" (UniqueName: \"kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hjx7g\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.503300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.503263 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:42.521721 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:42.521695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:42.967329 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.967292 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:42.972375 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:42.972343 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:43.656613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.656580 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:17:43.672274 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.672242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.673693 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.673661 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:17:43.763461 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.763425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk28j\" (UniqueName: \"kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.763689 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.763481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.864376 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.864345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk28j\" (UniqueName: \"kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.864550 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.864388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.864757 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.864737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.877291 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.877262 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk28j\" (UniqueName: \"kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j\") pod \"kuadrant-operator-controller-manager-86878c769f-zjmbd\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:43.986598 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:43.986489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:44.081549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:44.081521 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-92bss" Jul 10 13:17:54.780915 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.767520 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:54.780915 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:54.778920 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:54.818055 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.818029 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:54.820353 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:54.820320 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:54.822509 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:54.822479 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:54.852361 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.852340 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r"] Jul 10 13:17:54.854757 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:54.854729 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode549be6b_ab7b_4a33_82f2_9bcfb91272ef.slice/crio-e586ba8e19a3eb1fdedbcd704d79732df27054211d162e0180ba024d131b8535 WatchSource:0}: Error finding container e586ba8e19a3eb1fdedbcd704d79732df27054211d162e0180ba024d131b8535: Status 404 returned error can't find the container with id e586ba8e19a3eb1fdedbcd704d79732df27054211d162e0180ba024d131b8535 Jul 10 13:17:54.867806 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.867787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dgvq\" (UniqueName: \"kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq\") pod \"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5\" (UID: \"3ca937c9-eeb8-4fc1-8cb0-add95b6529d5\") " Jul 10 13:17:54.867889 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.867824 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume\") pod \"062b79f3-6ad3-4816-969b-3348451f6c31\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " Jul 10 13:17:54.867889 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.867878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvt9n\" (UniqueName: \"kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n\") pod \"062b79f3-6ad3-4816-969b-3348451f6c31\" (UID: \"062b79f3-6ad3-4816-969b-3348451f6c31\") " Jul 10 13:17:54.868347 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.868321 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "062b79f3-6ad3-4816-969b-3348451f6c31" (UID: "062b79f3-6ad3-4816-969b-3348451f6c31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:54.869939 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.869915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n" (OuterVolumeSpecName: "kube-api-access-zvt9n") pod "062b79f3-6ad3-4816-969b-3348451f6c31" (UID: "062b79f3-6ad3-4816-969b-3348451f6c31"). InnerVolumeSpecName "kube-api-access-zvt9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:54.870010 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.869923 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq" (OuterVolumeSpecName: "kube-api-access-6dgvq") pod "3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" (UID: "3ca937c9-eeb8-4fc1-8cb0-add95b6529d5"). InnerVolumeSpecName "kube-api-access-6dgvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:54.968400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.968372 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgvq\" (UniqueName: \"kubernetes.io/projected/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5-kube-api-access-6dgvq\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:17:54.968400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.968394 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062b79f3-6ad3-4816-969b-3348451f6c31-extensions-socket-volume\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:17:54.968400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.968403 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvt9n\" (UniqueName: \"kubernetes.io/projected/062b79f3-6ad3-4816-969b-3348451f6c31-kube-api-access-zvt9n\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:17:54.969078 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.969055 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" path="/var/lib/kubelet/pods/062b79f3-6ad3-4816-969b-3348451f6c31/volumes" Jul 10 13:17:54.969377 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:54.969364 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" path="/var/lib/kubelet/pods/3ca937c9-eeb8-4fc1-8cb0-add95b6529d5/volumes" Jul 10 13:17:55.061244 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.061211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:17:55.065046 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:55.065017 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d68e0e_2358_4f89_8d08_c3d5de9dafac.slice/crio-f37d06ed8a7491a160a2cc1c3d9662d38eb851190f8a58fda2bc6d2cdecc8a91 WatchSource:0}: Error finding container f37d06ed8a7491a160a2cc1c3d9662d38eb851190f8a58fda2bc6d2cdecc8a91: Status 404 returned error can't find the container with id f37d06ed8a7491a160a2cc1c3d9662d38eb851190f8a58fda2bc6d2cdecc8a91 Jul 10 13:17:55.067121 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.067100 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:17:55.069950 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:17:55.069929 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda034de6b_ca71_4140_993a_d7c477719036.slice/crio-6fa7844fcfc03c8482a8f1b52b598e2fff6d99ec82b66e9b53b68443df5605c5 WatchSource:0}: Error finding container 6fa7844fcfc03c8482a8f1b52b598e2fff6d99ec82b66e9b53b68443df5605c5: Status 404 returned error can't find the container with id 6fa7844fcfc03c8482a8f1b52b598e2fff6d99ec82b66e9b53b68443df5605c5 Jul 10 13:17:55.193322 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.193286 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" containerID="5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d" exitCode=0 Jul 10 13:17:55.193470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.193328 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:55.193470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.193365 2573 scope.go:117] "RemoveContainer" containerID="5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d" Jul 10 13:17:55.195483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.195456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" event={"ID":"c8d68e0e-2358-4f89-8d08-c3d5de9dafac","Type":"ContainerStarted","Data":"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee"} Jul 10 13:17:55.195613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.195512 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" event={"ID":"c8d68e0e-2358-4f89-8d08-c3d5de9dafac","Type":"ContainerStarted","Data":"f37d06ed8a7491a160a2cc1c3d9662d38eb851190f8a58fda2bc6d2cdecc8a91"} Jul 10 13:17:55.195613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.195543 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:17:55.195938 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.195913 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:55.197139 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.197093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" event={"ID":"d42c7116-492f-4386-b009-388cf862866c","Type":"ContainerStarted","Data":"6c78fc7d1990b3e95d0f920e0ca4ee4c1a71552689d17190ca9a493c907e6d33"} Jul 10 13:17:55.198077 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.198052 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:55.198774 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.198755 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" event={"ID":"a034de6b-ca71-4140-993a-d7c477719036","Type":"ContainerStarted","Data":"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a"} Jul 10 13:17:55.198875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.198794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:17:55.198875 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.198816 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" event={"ID":"a034de6b-ca71-4140-993a-d7c477719036","Type":"ContainerStarted","Data":"6fa7844fcfc03c8482a8f1b52b598e2fff6d99ec82b66e9b53b68443df5605c5"} Jul 10 13:17:55.200347 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.200326 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" event={"ID":"e549be6b-ab7b-4a33-82f2-9bcfb91272ef","Type":"ContainerStarted","Data":"80747ce3dde097c65e9c66204c5b9c1c7c568e5030b3e41d4678a33e9f9837df"} Jul 10 13:17:55.200438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.200360 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" event={"ID":"e549be6b-ab7b-4a33-82f2-9bcfb91272ef","Type":"ContainerStarted","Data":"e586ba8e19a3eb1fdedbcd704d79732df27054211d162e0180ba024d131b8535"} Jul 10 13:17:55.200438 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.200426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:17:55.201702 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.201677 2573 generic.go:356] "Generic (PLEG): container finished" podID="062b79f3-6ad3-4816-969b-3348451f6c31" containerID="2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a" exitCode=0 Jul 10 13:17:55.201792 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.201709 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:55.202876 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.202857 2573 scope.go:117] "RemoveContainer" containerID="5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d" Jul 10 13:17:55.203477 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.203089 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d\": container with ID starting with 5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d not found: ID does not exist" containerID="5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d" Jul 10 13:17:55.203477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.203125 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d"} err="failed to get container status \"5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d\": rpc error: code = NotFound desc = could not find container \"5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d\": container with ID starting with 5e9a7a93d1c3165750c48eba6bb8b30d2959785e6c1c295fb7a8f7b2c613261d not found: ID does not exist" Jul 10 13:17:55.203477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.203143 2573 scope.go:117] "RemoveContainer" containerID="2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a" Jul 10 13:17:55.210932 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.210918 2573 scope.go:117] "RemoveContainer" containerID="2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a" Jul 10 13:17:55.211165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.211147 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a\": container with ID starting with 2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a not found: ID does not exist" containerID="2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a" Jul 10 13:17:55.211227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.211177 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a"} err="failed to get container status \"2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a\": rpc error: code = NotFound desc = could not find container \"2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a\": container with ID starting with 2675c23c3673859b5dc8ac66c6fd404d9527b1ab05038b5b81a1b053690af35a not found: ID does not exist" Jul 10 13:17:55.221113 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.221074 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wmczd\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="062b79f3-6ad3-4816-969b-3348451f6c31" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wmczd" Jul 10 13:17:55.221420 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.221387 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" podStartSLOduration=12.221375098 podStartE2EDuration="12.221375098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:55.219141305 +0000 UTC m=+472.751743605" watchObservedRunningTime="2026-07-10 13:17:55.221375098 +0000 UTC m=+472.753977396" Jul 10 13:17:55.250872 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.250828 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" podStartSLOduration=13.250814378 podStartE2EDuration="13.250814378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:55.249280545 +0000 UTC m=+472.781882846" watchObservedRunningTime="2026-07-10 13:17:55.250814378 +0000 UTC m=+472.783416677" Jul 10 13:17:55.251749 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:17:55.251726 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gzgn5\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="3ca937c9-eeb8-4fc1-8cb0-add95b6529d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gzgn5" Jul 10 13:17:55.283435 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.283389 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvtss" podStartSLOduration=2.060724607 podStartE2EDuration="24.283375316s" totalImagesPullingTime="22.222650709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:55.283086962 +0000 UTC m=+472.815689261" watchObservedRunningTime="2026-07-10 13:17:55.283375316 +0000 UTC m=+472.815977615" Jul 10 13:17:55.322847 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:17:55.322802 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" podStartSLOduration=13.322785737 podStartE2EDuration="13.322785737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:17:55.320848466 +0000 UTC m=+472.853450776" watchObservedRunningTime="2026-07-10 13:17:55.322785737 +0000 UTC m=+472.855388039" Jul 10 13:18:06.212217 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.212183 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l5t8r" Jul 10 13:18:06.212764 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.212272 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:18:06.212764 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.212436 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:18:06.305206 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.305174 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:18:06.305418 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.305395 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" podUID="a034de6b-ca71-4140-993a-d7c477719036" containerName="manager" containerID="cri-o://e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a" gracePeriod=10 Jul 10 13:18:06.557458 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.557431 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:18:06.660488 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.660464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfwrz\" (UniqueName: \"kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz\") pod \"a034de6b-ca71-4140-993a-d7c477719036\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " Jul 10 13:18:06.660626 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.660521 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume\") pod \"a034de6b-ca71-4140-993a-d7c477719036\" (UID: \"a034de6b-ca71-4140-993a-d7c477719036\") " Jul 10 13:18:06.660837 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.660813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a034de6b-ca71-4140-993a-d7c477719036" (UID: "a034de6b-ca71-4140-993a-d7c477719036"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:18:06.662456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.662433 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz" (OuterVolumeSpecName: "kube-api-access-lfwrz") pod "a034de6b-ca71-4140-993a-d7c477719036" (UID: "a034de6b-ca71-4140-993a-d7c477719036"). InnerVolumeSpecName "kube-api-access-lfwrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:18:06.761466 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.761409 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwrz\" (UniqueName: \"kubernetes.io/projected/a034de6b-ca71-4140-993a-d7c477719036-kube-api-access-lfwrz\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:18:06.761466 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:06.761430 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a034de6b-ca71-4140-993a-d7c477719036-extensions-socket-volume\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:18:07.265697 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.265664 2573 generic.go:356] "Generic (PLEG): container finished" podID="a034de6b-ca71-4140-993a-d7c477719036" containerID="e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a" exitCode=0 Jul 10 13:18:07.266096 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.265717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" event={"ID":"a034de6b-ca71-4140-993a-d7c477719036","Type":"ContainerDied","Data":"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a"} Jul 10 13:18:07.266096 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.265757 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" Jul 10 13:18:07.266096 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.265773 2573 scope.go:117] "RemoveContainer" containerID="e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a" Jul 10 13:18:07.266096 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.265763 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g" event={"ID":"a034de6b-ca71-4140-993a-d7c477719036","Type":"ContainerDied","Data":"6fa7844fcfc03c8482a8f1b52b598e2fff6d99ec82b66e9b53b68443df5605c5"} Jul 10 13:18:07.274216 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.274161 2573 scope.go:117] "RemoveContainer" containerID="e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a" Jul 10 13:18:07.274533 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:18:07.274506 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a\": container with ID starting with e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a not found: ID does not exist" containerID="e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a" Jul 10 13:18:07.274663 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.274544 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a"} err="failed to get container status \"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a\": rpc error: code = NotFound desc = could not find container \"e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a\": container with ID starting with e9190777f366c8da4f67b8ba1a6c5531a67eda7263ee4961ae0da1ddb413a99a not found: ID does not exist" Jul 10 13:18:07.295747 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.295727 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:18:07.300007 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:07.299987 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hjx7g"] Jul 10 13:18:08.974687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:08.974650 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a034de6b-ca71-4140-993a-d7c477719036" path="/var/lib/kubelet/pods/a034de6b-ca71-4140-993a-d7c477719036/volumes" Jul 10 13:18:26.092925 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.092884 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:18:26.093476 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.093456 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a034de6b-ca71-4140-993a-d7c477719036" containerName="manager" Jul 10 13:18:26.093568 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.093479 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a034de6b-ca71-4140-993a-d7c477719036" containerName="manager" Jul 10 13:18:26.093568 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.093564 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a034de6b-ca71-4140-993a-d7c477719036" containerName="manager" Jul 10 13:18:26.173537 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.173508 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:18:26.173719 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.173547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:18:26.173719 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.173655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.179737 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.179695 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 13:18:26.328088 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.328055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxlh\" (UniqueName: \"kubernetes.io/projected/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-kube-api-access-xfxlh\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.328251 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.328180 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-config-file\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.429336 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.429310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxlh\" (UniqueName: \"kubernetes.io/projected/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-kube-api-access-xfxlh\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.429536 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.429380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-config-file\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.429949 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.429931 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-config-file\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.446488 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.446461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxlh\" (UniqueName: \"kubernetes.io/projected/d3e5f8c9-7133-45f1-93f0-07c9efa077a6-kube-api-access-xfxlh\") pod \"limitador-limitador-54d6d988d5-vvwxd\" (UID: \"d3e5f8c9-7133-45f1-93f0-07c9efa077a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.484253 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.484234 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:26.637429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:26.637398 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:18:26.639312 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:18:26.639283 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e5f8c9_7133_45f1_93f0_07c9efa077a6.slice/crio-86e8a345ff24487e8e085281aae0edfb99a2cce1840abb31c38ddf79e984aaca WatchSource:0}: Error finding container 86e8a345ff24487e8e085281aae0edfb99a2cce1840abb31c38ddf79e984aaca: Status 404 returned error can't find the container with id 86e8a345ff24487e8e085281aae0edfb99a2cce1840abb31c38ddf79e984aaca Jul 10 13:18:27.370123 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:27.370085 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" event={"ID":"d3e5f8c9-7133-45f1-93f0-07c9efa077a6","Type":"ContainerStarted","Data":"86e8a345ff24487e8e085281aae0edfb99a2cce1840abb31c38ddf79e984aaca"} Jul 10 13:18:29.381227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:29.381193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" event={"ID":"d3e5f8c9-7133-45f1-93f0-07c9efa077a6","Type":"ContainerStarted","Data":"988cbcaa7d7f34c495369878a99e0fcb19a6204ad7fd1af76f5931612b24ad29"} Jul 10 13:18:29.381676 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:29.381287 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:18:29.420467 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:29.420384 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" podStartSLOduration=0.912540431 podStartE2EDuration="3.420370821s" totalImagesPullingTime="2.50783039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:18:29.416711931 +0000 UTC m=+506.949314243" watchObservedRunningTime="2026-07-10 13:18:29.420370821 +0000 UTC m=+506.952973119" Jul 10 13:18:40.386934 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:18:40.386904 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwxd" Jul 10 13:19:42.427532 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:19:42.427490 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:20:02.085641 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:02.085593 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:20:02.894083 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:02.894054 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:20:02.897254 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:02.897231 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:20:16.682176 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:16.682144 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:20:22.302283 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:22.302250 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:20:36.642702 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:20:36.642581 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:21:07.233566 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:21:07.233532 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:21:16.891241 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:21:16.891211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:22:22.396606 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:22:22.396536 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:22:31.483307 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:22:31.483271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:22:41.274615 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:22:41.274577 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:22:51.189245 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:22:51.189211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:23:00.490435 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:23:00.490400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:23:11.831270 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:23:11.831235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:23:35.987485 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:23:35.987403 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:23:47.932774 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:23:47.932740 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:24:24.997494 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:24:24.997458 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:24:34.584986 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:24:34.584955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:24:43.691676 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:24:43.691624 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:24:55.186909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:24:55.186823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:25:02.924795 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:02.924763 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:25:02.925526 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:02.925507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:25:06.087696 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:06.087666 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:25:15.317477 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:15.317442 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:25:25.784933 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:25.784904 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:25:35.780608 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:25:35.780575 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:26:00.683870 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:26:00.683827 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:26:12.290030 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:26:12.290000 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:26:36.781279 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:26:36.781189 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:26:47.399253 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:26:47.399224 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:27:02.887120 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:27:02.887082 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:27:14.294077 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:27:14.294045 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:27:24.078158 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:27:24.078125 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:27:35.785099 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:27:35.785068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:28:07.520183 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:07.520093 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:28:07.520869 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:07.520297 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" podUID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" containerName="manager" containerID="cri-o://42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee" gracePeriod=10 Jul 10 13:28:10.878367 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:10.878338 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:28:10.904162 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:10.904136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume\") pod \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " Jul 10 13:28:10.904311 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:10.904193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk28j\" (UniqueName: \"kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j\") pod \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\" (UID: \"c8d68e0e-2358-4f89-8d08-c3d5de9dafac\") " Jul 10 13:28:10.904554 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:10.904521 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c8d68e0e-2358-4f89-8d08-c3d5de9dafac" (UID: "c8d68e0e-2358-4f89-8d08-c3d5de9dafac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:28:10.906422 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:10.906396 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j" (OuterVolumeSpecName: "kube-api-access-gk28j") pod "c8d68e0e-2358-4f89-8d08-c3d5de9dafac" (UID: "c8d68e0e-2358-4f89-8d08-c3d5de9dafac"). InnerVolumeSpecName "kube-api-access-gk28j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:28:11.005204 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.005146 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk28j\" (UniqueName: \"kubernetes.io/projected/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-kube-api-access-gk28j\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:28:11.005204 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.005169 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8d68e0e-2358-4f89-8d08-c3d5de9dafac-extensions-socket-volume\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:28:11.208229 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.208198 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" containerID="42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee" exitCode=0 Jul 10 13:28:11.208374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.208244 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" Jul 10 13:28:11.208374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.208281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" event={"ID":"c8d68e0e-2358-4f89-8d08-c3d5de9dafac","Type":"ContainerDied","Data":"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee"} Jul 10 13:28:11.208374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.208337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd" event={"ID":"c8d68e0e-2358-4f89-8d08-c3d5de9dafac","Type":"ContainerDied","Data":"f37d06ed8a7491a160a2cc1c3d9662d38eb851190f8a58fda2bc6d2cdecc8a91"} Jul 10 13:28:11.208374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.208364 2573 scope.go:117] "RemoveContainer" containerID="42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee" Jul 10 13:28:11.217685 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.217669 2573 scope.go:117] "RemoveContainer" containerID="42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee" Jul 10 13:28:11.217962 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:28:11.217944 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee\": container with ID starting with 42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee not found: ID does not exist" containerID="42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee" Jul 10 13:28:11.218031 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.217975 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee"} err="failed to get container status \"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee\": rpc error: code = NotFound desc = could not find container \"42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee\": container with ID starting with 42e0060693a0cddf85b9ac024f90e89131e284b416135e8a3f6a7a330032d6ee not found: ID does not exist" Jul 10 13:28:11.224130 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.224108 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:28:11.227742 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:11.227722 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zjmbd"] Jul 10 13:28:12.968346 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:28:12.968312 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" path="/var/lib/kubelet/pods/c8d68e0e-2358-4f89-8d08-c3d5de9dafac/volumes" Jul 10 13:29:08.779189 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.779153 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n"] Jul 10 13:29:08.779561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.779522 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" containerName="manager" Jul 10 13:29:08.779561 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.779534 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" containerName="manager" Jul 10 13:29:08.779645 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.779602 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d68e0e-2358-4f89-8d08-c3d5de9dafac" containerName="manager" Jul 10 13:29:08.783778 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.783761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.788524 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.788504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmfzh\"" Jul 10 13:29:08.799091 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.799072 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n"] Jul 10 13:29:08.861320 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.861290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjk2h\" (UniqueName: \"kubernetes.io/projected/7474bd25-817a-4dfe-9d72-e9c734120c49-kube-api-access-gjk2h\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.861468 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.861351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7474bd25-817a-4dfe-9d72-e9c734120c49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.962275 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.962243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7474bd25-817a-4dfe-9d72-e9c734120c49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.962446 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.962301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjk2h\" (UniqueName: \"kubernetes.io/projected/7474bd25-817a-4dfe-9d72-e9c734120c49-kube-api-access-gjk2h\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.962720 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.962695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7474bd25-817a-4dfe-9d72-e9c734120c49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:08.971889 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:08.971867 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjk2h\" (UniqueName: \"kubernetes.io/projected/7474bd25-817a-4dfe-9d72-e9c734120c49-kube-api-access-gjk2h\") pod \"kuadrant-operator-controller-manager-86878c769f-qgx7n\" (UID: \"7474bd25-817a-4dfe-9d72-e9c734120c49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:09.093728 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.093664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:09.228648 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.228594 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n"] Jul 10 13:29:09.233009 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:29:09.232983 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7474bd25_817a_4dfe_9d72_e9c734120c49.slice/crio-512c02ee6eaf9b1eb7119816561967f855d049c54a4a8fa85f9da82da2833ee9 WatchSource:0}: Error finding container 512c02ee6eaf9b1eb7119816561967f855d049c54a4a8fa85f9da82da2833ee9: Status 404 returned error can't find the container with id 512c02ee6eaf9b1eb7119816561967f855d049c54a4a8fa85f9da82da2833ee9 Jul 10 13:29:09.479246 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.479214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" event={"ID":"7474bd25-817a-4dfe-9d72-e9c734120c49","Type":"ContainerStarted","Data":"38b0341ca210129070375ff48c71e910efdc555ceaab241f4407357375768b73"} Jul 10 13:29:09.479402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.479264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" event={"ID":"7474bd25-817a-4dfe-9d72-e9c734120c49","Type":"ContainerStarted","Data":"512c02ee6eaf9b1eb7119816561967f855d049c54a4a8fa85f9da82da2833ee9"} Jul 10 13:29:09.479402 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.479374 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:09.502623 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:09.502578 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" podStartSLOduration=1.5025643720000001 podStartE2EDuration="1.502564372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:29:09.500374308 +0000 UTC m=+1147.032976610" watchObservedRunningTime="2026-07-10 13:29:09.502564372 +0000 UTC m=+1147.035166673" Jul 10 13:29:20.487155 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:20.487118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qgx7n" Jul 10 13:29:30.989595 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:30.989510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:29:39.581894 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:39.581862 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:29:53.987654 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:29:53.987598 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:00.145618 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.145581 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:30:00.149518 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.149500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" Jul 10 13:30:00.152508 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.152485 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hj9h2\"" Jul 10 13:30:00.158946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.158926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" Jul 10 13:30:00.165362 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.165339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:30:00.300270 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.300242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:30:00.301962 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:30:00.301939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7660d0_4dff_4d72_a64c_2e47ba3a115b.slice/crio-2fac04608654e38d5ab5699dd0b26435a54d50d136908035b4d8c39f381c39d4 WatchSource:0}: Error finding container 2fac04608654e38d5ab5699dd0b26435a54d50d136908035b4d8c39f381c39d4: Status 404 returned error can't find the container with id 2fac04608654e38d5ab5699dd0b26435a54d50d136908035b4d8c39f381c39d4 Jul 10 13:30:00.304018 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.303997 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:30:00.723488 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:00.723452 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" event={"ID":"6d7660d0-4dff-4d72-a64c-2e47ba3a115b","Type":"ContainerStarted","Data":"2fac04608654e38d5ab5699dd0b26435a54d50d136908035b4d8c39f381c39d4"} Jul 10 13:30:02.954713 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:02.954687 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:30:02.961765 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:02.961740 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:30:05.491502 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:05.491466 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:13.794090 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:13.794060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" event={"ID":"6d7660d0-4dff-4d72-a64c-2e47ba3a115b","Type":"ContainerStarted","Data":"bf9746122599e1d02e377352db4a3d7de4879ebd0ac1135d99e3afb039942088"} Jul 10 13:30:13.811983 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:13.811910 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" podStartSLOduration=0.555160018 podStartE2EDuration="13.811896178s" totalImagesPullingTime="13.25673616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:30:13.810720599 +0000 UTC m=+1211.343322897" watchObservedRunningTime="2026-07-10 13:30:13.811896178 +0000 UTC m=+1211.344498477" Jul 10 13:30:17.025325 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:17.025286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:26.987176 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:26.987139 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:36.287199 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:36.287162 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:47.202996 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:47.202948 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:30:58.696704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:30:58.696606 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:31:00.011925 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:00.011889 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:31:00.012364 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:00.012106 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" podUID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" containerName="cleanup" containerID="cri-o://bf9746122599e1d02e377352db4a3d7de4879ebd0ac1135d99e3afb039942088" gracePeriod=30 Jul 10 13:31:07.283799 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:07.283761 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:31:18.494921 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:18.494887 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:31:30.162007 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:30.161974 2573 generic.go:356] "Generic (PLEG): container finished" podID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" containerID="bf9746122599e1d02e377352db4a3d7de4879ebd0ac1135d99e3afb039942088" exitCode=137 Jul 10 13:31:30.162346 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:30.162045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" event={"ID":"6d7660d0-4dff-4d72-a64c-2e47ba3a115b","Type":"ContainerDied","Data":"bf9746122599e1d02e377352db4a3d7de4879ebd0ac1135d99e3afb039942088"} Jul 10 13:31:30.188627 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:30.188606 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" Jul 10 13:31:31.168360 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:31.168327 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" Jul 10 13:31:31.168831 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:31.168327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf" event={"ID":"6d7660d0-4dff-4d72-a64c-2e47ba3a115b","Type":"ContainerDied","Data":"2fac04608654e38d5ab5699dd0b26435a54d50d136908035b4d8c39f381c39d4"} Jul 10 13:31:31.168831 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:31.168459 2573 scope.go:117] "RemoveContainer" containerID="bf9746122599e1d02e377352db4a3d7de4879ebd0ac1135d99e3afb039942088" Jul 10 13:31:31.184366 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:31.184342 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:31:31.190356 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:31.190334 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728170-8kfkf"] Jul 10 13:31:32.969132 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:32.969098 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" path="/var/lib/kubelet/pods/6d7660d0-4dff-4d72-a64c-2e47ba3a115b/volumes" Jul 10 13:31:40.600822 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:40.600786 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:31:49.589580 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:49.589546 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:31:59.984413 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:31:59.984377 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:32:10.797999 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:32:10.797965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:32:18.786210 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:32:18.786178 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:32:28.200168 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:32:28.200092 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:32:41.218046 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:32:41.218013 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:32:52.627254 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:32:52.627209 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:00.588208 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:00.588173 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:10.093234 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:10.093197 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:19.896671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:19.896623 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:33.024396 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:33.024357 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:42.545841 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:42.545807 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:33:52.294791 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:33:52.294756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:34:00.903417 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:34:00.903339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:34:10.587106 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:34:10.587069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:34:20.988854 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:34:20.988826 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:34:38.589268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:34:38.589231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:34:50.722446 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:34:50.722391 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:01.877936 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:01.877900 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:02.989238 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:02.989210 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:35:02.991568 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:02.991537 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:35:15.281274 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.281238 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:15.282080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.282059 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" containerName="cleanup" Jul 10 13:35:15.282080 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.282082 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" containerName="cleanup" Jul 10 13:35:15.282224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.282190 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7660d0-4dff-4d72-a64c-2e47ba3a115b" containerName="cleanup" Jul 10 13:35:15.285341 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.285321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.288067 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.288039 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:35:15.288298 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.288280 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:35:15.288360 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.288329 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2a72c3fa-serving-cert\"" Jul 10 13:35:15.288697 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.288676 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:35:15.298892 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.298865 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:15.337226 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.337196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.337395 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.337234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.337395 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.337316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrdh\" (UniqueName: \"kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.438054 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.438019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.438243 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.438062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.438243 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.438121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrdh\" (UniqueName: \"kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.438378 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:15.438249 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-serving-cert: secret "maas-api-e2e-ait-2a72c3fa-serving-cert" not found Jul 10 13:35:15.438378 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:15.438326 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls podName:f3f3ff82-cce8-4716-8a7e-0a212ed1431c nodeName:}" failed. No retries permitted until 2026-07-10 13:35:15.938307915 +0000 UTC m=+1513.470910194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls") pod "maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" (UID: "f3f3ff82-cce8-4716-8a7e-0a212ed1431c") : secret "maas-api-e2e-ait-2a72c3fa-serving-cert" not found Jul 10 13:35:15.438756 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.438735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.450151 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.450126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrdh\" (UniqueName: \"kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.942325 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.942288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:15.944811 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:15.944789 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") pod \"maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:16.198503 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:16.198417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:16.341443 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:16.341312 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:16.344285 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:35:16.344248 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f3ff82_cce8_4716_8a7e_0a212ed1431c.slice/crio-68529efbccc93e50614a701611c301444bce9be6f4157821bd682e49e321705b WatchSource:0}: Error finding container 68529efbccc93e50614a701611c301444bce9be6f4157821bd682e49e321705b: Status 404 returned error can't find the container with id 68529efbccc93e50614a701611c301444bce9be6f4157821bd682e49e321705b Jul 10 13:35:16.345475 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:16.345458 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:35:17.261997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:17.261957 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" event={"ID":"f3f3ff82-cce8-4716-8a7e-0a212ed1431c","Type":"ContainerStarted","Data":"68529efbccc93e50614a701611c301444bce9be6f4157821bd682e49e321705b"} Jul 10 13:35:18.188585 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:18.188544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:19.274243 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:19.274155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" event={"ID":"f3f3ff82-cce8-4716-8a7e-0a212ed1431c","Type":"ContainerStarted","Data":"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f"} Jul 10 13:35:19.274680 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:19.274286 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:19.307088 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:19.307041 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" podStartSLOduration=1.739307512 podStartE2EDuration="4.307026759s" totalImagesPullingTime="2.567719247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:19.30380009 +0000 UTC m=+1516.836402388" watchObservedRunningTime="2026-07-10 13:35:19.307026759 +0000 UTC m=+1516.839629085" Jul 10 13:35:20.651427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:20.651388 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:21.283331 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.283285 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" podUID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" containerName="maas-api" containerID="cri-o://050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f" gracePeriod=30 Jul 10 13:35:21.541752 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.541697 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:21.590022 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.589992 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") pod \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " Jul 10 13:35:21.590142 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.590086 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca\") pod \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " Jul 10 13:35:21.590142 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.590111 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwrdh\" (UniqueName: \"kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh\") pod \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\" (UID: \"f3f3ff82-cce8-4716-8a7e-0a212ed1431c\") " Jul 10 13:35:21.590451 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.590427 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3f3ff82-cce8-4716-8a7e-0a212ed1431c" (UID: "f3f3ff82-cce8-4716-8a7e-0a212ed1431c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:21.592047 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.592025 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3f3ff82-cce8-4716-8a7e-0a212ed1431c" (UID: "f3f3ff82-cce8-4716-8a7e-0a212ed1431c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:35:21.592146 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.592116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh" (OuterVolumeSpecName: "kube-api-access-rwrdh") pod "f3f3ff82-cce8-4716-8a7e-0a212ed1431c" (UID: "f3f3ff82-cce8-4716-8a7e-0a212ed1431c"). InnerVolumeSpecName "kube-api-access-rwrdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:21.691423 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.691385 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:21.691423 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.691412 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrdh\" (UniqueName: \"kubernetes.io/projected/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-kube-api-access-rwrdh\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:21.691423 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:21.691422 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f3ff82-cce8-4716-8a7e-0a212ed1431c-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:22.289908 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.289873 2573 generic.go:356] "Generic (PLEG): container finished" podID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" containerID="050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f" exitCode=0 Jul 10 13:35:22.290089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.289915 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" Jul 10 13:35:22.290089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.289917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" event={"ID":"f3f3ff82-cce8-4716-8a7e-0a212ed1431c","Type":"ContainerDied","Data":"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f"} Jul 10 13:35:22.290089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.290036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8" event={"ID":"f3f3ff82-cce8-4716-8a7e-0a212ed1431c","Type":"ContainerDied","Data":"68529efbccc93e50614a701611c301444bce9be6f4157821bd682e49e321705b"} Jul 10 13:35:22.290089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.290060 2573 scope.go:117] "RemoveContainer" containerID="050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f" Jul 10 13:35:22.299538 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.299514 2573 scope.go:117] "RemoveContainer" containerID="050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f" Jul 10 13:35:22.299807 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:22.299789 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f\": container with ID starting with 050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f not found: ID does not exist" containerID="050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f" Jul 10 13:35:22.299867 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.299817 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f"} err="failed to get container status \"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f\": rpc error: code = NotFound desc = could not find container \"050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f\": container with ID starting with 050b9c81fce2f82bd252892fe815a7a7ba4e4ba48e32535b20d3f14c9dc4bf4f not found: ID does not exist" Jul 10 13:35:22.311686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.311655 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:22.318173 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.318149 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a72c3fa-6bcd9cf78d-5h7n8"] Jul 10 13:35:22.971181 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:22.971144 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" path="/var/lib/kubelet/pods/f3f3ff82-cce8-4716-8a7e-0a212ed1431c/volumes" Jul 10 13:35:27.735842 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.735812 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:27.736295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.736205 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" containerName="maas-api" Jul 10 13:35:27.736295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.736217 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" containerName="maas-api" Jul 10 13:35:27.736295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.736295 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f3ff82-cce8-4716-8a7e-0a212ed1431c" containerName="maas-api" Jul 10 13:35:27.740807 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.740785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.743467 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.743095 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-28b6c6a0-serving-cert\"" Jul 10 13:35:27.743467 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.743137 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:35:27.743467 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.743306 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:35:27.743953 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.743939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:35:27.751021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.750997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:27.843858 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.843822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.844021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.843869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9c6\" (UniqueName: \"kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.844021 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.843921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.945189 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.945151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.945189 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.945197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9c6\" (UniqueName: \"kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.945436 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.945236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.945961 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.945942 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.947690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.947674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:27.954504 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:27.954480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9c6\" (UniqueName: \"kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6\") pod \"maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:28.054325 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:28.054250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:28.192536 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:28.192514 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:28.194840 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:35:28.194812 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec394318_5c42_49d8_bbeb_93f2790c00d3.slice/crio-1eb76cc35b4786ac6d6fd8d31e4b13146cc5a9e4edcf850b9068e92b1d99b80b WatchSource:0}: Error finding container 1eb76cc35b4786ac6d6fd8d31e4b13146cc5a9e4edcf850b9068e92b1d99b80b: Status 404 returned error can't find the container with id 1eb76cc35b4786ac6d6fd8d31e4b13146cc5a9e4edcf850b9068e92b1d99b80b Jul 10 13:35:28.320205 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:28.320133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" event={"ID":"ec394318-5c42-49d8-bbeb-93f2790c00d3","Type":"ContainerStarted","Data":"1eb76cc35b4786ac6d6fd8d31e4b13146cc5a9e4edcf850b9068e92b1d99b80b"} Jul 10 13:35:29.290671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:29.290623 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:29.327210 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:29.327167 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" event={"ID":"ec394318-5c42-49d8-bbeb-93f2790c00d3","Type":"ContainerStarted","Data":"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5"} Jul 10 13:35:29.327350 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:29.327284 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:29.350250 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:29.350203 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" podStartSLOduration=1.941116801 podStartE2EDuration="2.350189234s" totalImagesPullingTime="409.072433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:29.349040655 +0000 UTC m=+1526.881642953" watchObservedRunningTime="2026-07-10 13:35:29.350189234 +0000 UTC m=+1526.882791533" Jul 10 13:35:33.046975 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.046946 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:33.047578 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.047108 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" podUID="ec394318-5c42-49d8-bbeb-93f2790c00d3" containerName="maas-api" containerID="cri-o://1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5" gracePeriod=30 Jul 10 13:35:33.052417 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.052396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:33.299793 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.299736 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:33.353258 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.353223 2573 generic.go:356] "Generic (PLEG): container finished" podID="ec394318-5c42-49d8-bbeb-93f2790c00d3" containerID="1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5" exitCode=0 Jul 10 13:35:33.353429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.353259 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" event={"ID":"ec394318-5c42-49d8-bbeb-93f2790c00d3","Type":"ContainerDied","Data":"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5"} Jul 10 13:35:33.353429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.353279 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" Jul 10 13:35:33.353429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.353317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz" event={"ID":"ec394318-5c42-49d8-bbeb-93f2790c00d3","Type":"ContainerDied","Data":"1eb76cc35b4786ac6d6fd8d31e4b13146cc5a9e4edcf850b9068e92b1d99b80b"} Jul 10 13:35:33.353429 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.353346 2573 scope.go:117] "RemoveContainer" containerID="1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5" Jul 10 13:35:33.362815 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.362788 2573 scope.go:117] "RemoveContainer" containerID="1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5" Jul 10 13:35:33.363065 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:33.363048 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5\": container with ID starting with 1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5 not found: ID does not exist" containerID="1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5" Jul 10 13:35:33.363107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.363073 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5"} err="failed to get container status \"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5\": rpc error: code = NotFound desc = could not find container \"1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5\": container with ID starting with 1602a895061c9155b4b528b124f17ac00570f51d5969088b54f58cfdfd3a9ee5 not found: ID does not exist" Jul 10 13:35:33.394056 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.394033 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca\") pod \"ec394318-5c42-49d8-bbeb-93f2790c00d3\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " Jul 10 13:35:33.394137 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.394067 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz9c6\" (UniqueName: \"kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6\") pod \"ec394318-5c42-49d8-bbeb-93f2790c00d3\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " Jul 10 13:35:33.394182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.394146 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls\") pod \"ec394318-5c42-49d8-bbeb-93f2790c00d3\" (UID: \"ec394318-5c42-49d8-bbeb-93f2790c00d3\") " Jul 10 13:35:33.394413 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.394390 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec394318-5c42-49d8-bbeb-93f2790c00d3" (UID: "ec394318-5c42-49d8-bbeb-93f2790c00d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:33.396103 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.396075 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec394318-5c42-49d8-bbeb-93f2790c00d3" (UID: "ec394318-5c42-49d8-bbeb-93f2790c00d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:35:33.396176 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.396129 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6" (OuterVolumeSpecName: "kube-api-access-qz9c6") pod "ec394318-5c42-49d8-bbeb-93f2790c00d3" (UID: "ec394318-5c42-49d8-bbeb-93f2790c00d3"). InnerVolumeSpecName "kube-api-access-qz9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:33.495741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.495712 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec394318-5c42-49d8-bbeb-93f2790c00d3-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:33.495741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.495737 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9c6\" (UniqueName: \"kubernetes.io/projected/ec394318-5c42-49d8-bbeb-93f2790c00d3-kube-api-access-qz9c6\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:33.495741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.495747 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec394318-5c42-49d8-bbeb-93f2790c00d3-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:33.679994 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.679963 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:33.688921 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:33.688897 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28b6c6a0-cf6bc5947-p2gsz"] Jul 10 13:35:34.968512 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:34.968477 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec394318-5c42-49d8-bbeb-93f2790c00d3" path="/var/lib/kubelet/pods/ec394318-5c42-49d8-bbeb-93f2790c00d3/volumes" Jul 10 13:35:40.186886 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:40.186849 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:46.448263 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.448231 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:46.448639 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.448593 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec394318-5c42-49d8-bbeb-93f2790c00d3" containerName="maas-api" Jul 10 13:35:46.448639 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.448604 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec394318-5c42-49d8-bbeb-93f2790c00d3" containerName="maas-api" Jul 10 13:35:46.448720 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.448690 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec394318-5c42-49d8-bbeb-93f2790c00d3" containerName="maas-api" Jul 10 13:35:46.452793 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.452731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.456939 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.456917 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:35:46.457448 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.457430 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3763cabf-serving-cert\"" Jul 10 13:35:46.457547 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.457482 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:35:46.458383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.458368 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:35:46.469614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.469595 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:46.606279 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.606252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.606399 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.606285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.606399 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.606314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkjs\" (UniqueName: \"kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.707239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.707142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.707239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.707185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.707239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.707218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkjs\" (UniqueName: \"kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.707517 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:46.707317 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-serving-cert: secret "maas-api-e2e-derive-3763cabf-serving-cert" not found Jul 10 13:35:46.707517 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:46.707404 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls podName:1794678a-b0c3-4bbb-a31d-ced2b264e1ec nodeName:}" failed. No retries permitted until 2026-07-10 13:35:47.207382083 +0000 UTC m=+1544.739984360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls") pod "maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" (UID: "1794678a-b0c3-4bbb-a31d-ced2b264e1ec") : secret "maas-api-e2e-derive-3763cabf-serving-cert" not found Jul 10 13:35:46.707883 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.707862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:46.720551 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:46.720523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkjs\" (UniqueName: \"kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:47.212022 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:47.211992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:47.214457 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:47.214428 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") pod \"maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:47.363364 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:47.363337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:47.514756 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:47.514732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:47.517376 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:35:47.517350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1794678a_b0c3_4bbb_a31d_ced2b264e1ec.slice/crio-a937407dfa4d70147ce23b875732a861e8ea4693bc9cc588dfe46250460763cb WatchSource:0}: Error finding container a937407dfa4d70147ce23b875732a861e8ea4693bc9cc588dfe46250460763cb: Status 404 returned error can't find the container with id a937407dfa4d70147ce23b875732a861e8ea4693bc9cc588dfe46250460763cb Jul 10 13:35:48.429756 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:48.429715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" event={"ID":"1794678a-b0c3-4bbb-a31d-ced2b264e1ec","Type":"ContainerStarted","Data":"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f"} Jul 10 13:35:48.429933 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:48.429811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" event={"ID":"1794678a-b0c3-4bbb-a31d-ced2b264e1ec","Type":"ContainerStarted","Data":"a937407dfa4d70147ce23b875732a861e8ea4693bc9cc588dfe46250460763cb"} Jul 10 13:35:48.429933 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:48.429865 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:48.472063 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:48.472016 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" podStartSLOduration=2.114065885 podStartE2EDuration="2.472001792s" totalImagesPullingTime="357.935907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:48.470314757 +0000 UTC m=+1546.002917053" watchObservedRunningTime="2026-07-10 13:35:48.472001792 +0000 UTC m=+1546.004604090" Jul 10 13:35:49.490749 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:49.490710 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:35:50.989305 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:50.989273 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:50.989756 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:50.989453 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" podUID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" containerName="maas-api" containerID="cri-o://69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f" gracePeriod=30 Jul 10 13:35:51.247456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.247398 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:51.348224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.348196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca\") pod \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " Jul 10 13:35:51.348415 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.348263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfkjs\" (UniqueName: \"kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs\") pod \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " Jul 10 13:35:51.348415 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.348288 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") pod \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\" (UID: \"1794678a-b0c3-4bbb-a31d-ced2b264e1ec\") " Jul 10 13:35:51.348649 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.348605 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1794678a-b0c3-4bbb-a31d-ced2b264e1ec" (UID: "1794678a-b0c3-4bbb-a31d-ced2b264e1ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:51.350455 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.350426 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1794678a-b0c3-4bbb-a31d-ced2b264e1ec" (UID: "1794678a-b0c3-4bbb-a31d-ced2b264e1ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:35:51.350455 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.350441 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs" (OuterVolumeSpecName: "kube-api-access-pfkjs") pod "1794678a-b0c3-4bbb-a31d-ced2b264e1ec" (UID: "1794678a-b0c3-4bbb-a31d-ced2b264e1ec"). InnerVolumeSpecName "kube-api-access-pfkjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:51.443974 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.443945 2573 generic.go:356] "Generic (PLEG): container finished" podID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" containerID="69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f" exitCode=0 Jul 10 13:35:51.444094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.443988 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" Jul 10 13:35:51.444094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.444016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" event={"ID":"1794678a-b0c3-4bbb-a31d-ced2b264e1ec","Type":"ContainerDied","Data":"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f"} Jul 10 13:35:51.444094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.444059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw" event={"ID":"1794678a-b0c3-4bbb-a31d-ced2b264e1ec","Type":"ContainerDied","Data":"a937407dfa4d70147ce23b875732a861e8ea4693bc9cc588dfe46250460763cb"} Jul 10 13:35:51.444094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.444081 2573 scope.go:117] "RemoveContainer" containerID="69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f" Jul 10 13:35:51.448946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.448925 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:51.448946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.448946 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkjs\" (UniqueName: \"kubernetes.io/projected/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-kube-api-access-pfkjs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:51.449101 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.448957 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1794678a-b0c3-4bbb-a31d-ced2b264e1ec-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:35:51.452899 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.452881 2573 scope.go:117] "RemoveContainer" containerID="69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f" Jul 10 13:35:51.453165 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:35:51.453146 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f\": container with ID starting with 69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f not found: ID does not exist" containerID="69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f" Jul 10 13:35:51.453215 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.453175 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f"} err="failed to get container status \"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f\": rpc error: code = NotFound desc = could not find container \"69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f\": container with ID starting with 69826c2bb2fdfc2ea5e82162d030e94fa7685e599398b38d29208fbd133ff84f not found: ID does not exist" Jul 10 13:35:51.464715 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.464690 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:51.468767 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:51.468748 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3763cabf-65d45fd8cf-btwcw"] Jul 10 13:35:52.970897 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:52.970868 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" path="/var/lib/kubelet/pods/1794678a-b0c3-4bbb-a31d-ced2b264e1ec/volumes" Jul 10 13:35:58.588377 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:35:58.588344 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:37:19.705219 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:19.705145 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:37:28.885621 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:28.885586 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:37:36.838459 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.838429 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:36.838949 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.838821 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" containerName="maas-api" Jul 10 13:37:36.838949 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.838834 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" containerName="maas-api" Jul 10 13:37:36.838949 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.838920 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1794678a-b0c3-4bbb-a31d-ced2b264e1ec" containerName="maas-api" Jul 10 13:37:36.842173 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.842151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:36.844544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.844517 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-69fdb29c-serving-cert\"" Jul 10 13:37:36.844685 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.844586 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:37:36.844685 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.844613 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:37:36.845249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.845232 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:37:36.853446 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.853423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:36.937359 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.937335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4dm\" (UniqueName: \"kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:36.937516 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.937365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:36.937516 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:36.937388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.038027 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.038000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4dm\" (UniqueName: \"kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.038174 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.038030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.038174 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.038063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.038726 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.038704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.040409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.040390 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.049386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.049365 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4dm\" (UniqueName: \"kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm\") pod \"maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.154473 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.154450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.294130 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.294109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:37.296467 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:37:37.296438 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e55873_0125_4381_bc18_6021c914f743.slice/crio-fcb9aa81ea1f9852bebc77dd1704c0561f122e07909851c9b4cde3546f5dcf9a WatchSource:0}: Error finding container fcb9aa81ea1f9852bebc77dd1704c0561f122e07909851c9b4cde3546f5dcf9a: Status 404 returned error can't find the container with id fcb9aa81ea1f9852bebc77dd1704c0561f122e07909851c9b4cde3546f5dcf9a Jul 10 13:37:37.960826 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.960786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" event={"ID":"b5e55873-0125-4381-bc18-6021c914f743","Type":"ContainerStarted","Data":"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4"} Jul 10 13:37:37.961284 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.960875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" event={"ID":"b5e55873-0125-4381-bc18-6021c914f743","Type":"ContainerStarted","Data":"fcb9aa81ea1f9852bebc77dd1704c0561f122e07909851c9b4cde3546f5dcf9a"} Jul 10 13:37:37.961284 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.960970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:37.979690 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:37.979624 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" podStartSLOduration=1.491922689 podStartE2EDuration="1.979607944s" totalImagesPullingTime="487.685255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:37:37.977967364 +0000 UTC m=+1655.510569664" watchObservedRunningTime="2026-07-10 13:37:37.979607944 +0000 UTC m=+1655.512210245" Jul 10 13:37:43.970352 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:43.970324 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:43.986273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:43.986246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:37:49.046166 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.046134 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:49.046653 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.046395 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" podUID="b5e55873-0125-4381-bc18-6021c914f743" containerName="maas-api" containerID="cri-o://62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4" gracePeriod=30 Jul 10 13:37:49.294679 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.294654 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:49.343286 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.343211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx4dm\" (UniqueName: \"kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm\") pod \"b5e55873-0125-4381-bc18-6021c914f743\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " Jul 10 13:37:49.343286 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.343250 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca\") pod \"b5e55873-0125-4381-bc18-6021c914f743\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " Jul 10 13:37:49.343482 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.343309 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls\") pod \"b5e55873-0125-4381-bc18-6021c914f743\" (UID: \"b5e55873-0125-4381-bc18-6021c914f743\") " Jul 10 13:37:49.343614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.343591 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5e55873-0125-4381-bc18-6021c914f743" (UID: "b5e55873-0125-4381-bc18-6021c914f743"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:37:49.345257 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.345233 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm" (OuterVolumeSpecName: "kube-api-access-nx4dm") pod "b5e55873-0125-4381-bc18-6021c914f743" (UID: "b5e55873-0125-4381-bc18-6021c914f743"). InnerVolumeSpecName "kube-api-access-nx4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:37:49.345365 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.345312 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5e55873-0125-4381-bc18-6021c914f743" (UID: "b5e55873-0125-4381-bc18-6021c914f743"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:37:49.444845 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.444806 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e55873-0125-4381-bc18-6021c914f743-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:37:49.444845 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.444838 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4dm\" (UniqueName: \"kubernetes.io/projected/b5e55873-0125-4381-bc18-6021c914f743-kube-api-access-nx4dm\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:37:49.444845 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:49.444848 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e55873-0125-4381-bc18-6021c914f743-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:37:50.016800 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.016763 2573 generic.go:356] "Generic (PLEG): container finished" podID="b5e55873-0125-4381-bc18-6021c914f743" containerID="62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4" exitCode=0 Jul 10 13:37:50.016968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.016810 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" Jul 10 13:37:50.016968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.016851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" event={"ID":"b5e55873-0125-4381-bc18-6021c914f743","Type":"ContainerDied","Data":"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4"} Jul 10 13:37:50.016968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.016901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8" event={"ID":"b5e55873-0125-4381-bc18-6021c914f743","Type":"ContainerDied","Data":"fcb9aa81ea1f9852bebc77dd1704c0561f122e07909851c9b4cde3546f5dcf9a"} Jul 10 13:37:50.016968 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.016925 2573 scope.go:117] "RemoveContainer" containerID="62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4" Jul 10 13:37:50.026501 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.026480 2573 scope.go:117] "RemoveContainer" containerID="62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4" Jul 10 13:37:50.026800 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:37:50.026780 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4\": container with ID starting with 62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4 not found: ID does not exist" containerID="62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4" Jul 10 13:37:50.026863 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.026808 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4"} err="failed to get container status \"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4\": rpc error: code = NotFound desc = could not find container \"62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4\": container with ID starting with 62148ab05a886625806c0be6e0a555e66a7c0a2a414e648fd9638981792f07d4 not found: ID does not exist" Jul 10 13:37:50.042342 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.042322 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:50.048224 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.048195 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fdb29c-84997fb575-mg7z8"] Jul 10 13:37:50.969387 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:50.969352 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e55873-0125-4381-bc18-6021c914f743" path="/var/lib/kubelet/pods/b5e55873-0125-4381-bc18-6021c914f743/volumes" Jul 10 13:37:52.583662 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:37:52.583620 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:38:18.045801 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.045768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:38:18.046332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.046317 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e55873-0125-4381-bc18-6021c914f743" containerName="maas-api" Jul 10 13:38:18.046405 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.046334 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e55873-0125-4381-bc18-6021c914f743" containerName="maas-api" Jul 10 13:38:18.046460 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.046422 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e55873-0125-4381-bc18-6021c914f743" containerName="maas-api" Jul 10 13:38:18.049741 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.049720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.052495 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.052476 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:38:18.052606 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.052478 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:38:18.053215 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.053198 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:38:18.053312 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.053201 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-15039e79-serving-cert\"" Jul 10 13:38:18.062890 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.062867 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:38:18.183145 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.183110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.183331 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.183151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b46x\" (UniqueName: \"kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.183331 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.183242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.283811 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.283779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.283811 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.283811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b46x\" (UniqueName: \"kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.284000 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.283857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.284000 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:38:18.283957 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-serving-cert: secret "maas-api-e2e-mt-15039e79-serving-cert" not found Jul 10 13:38:18.284071 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:38:18.284035 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls podName:7132ca6b-28c2-453c-b4c9-0bd5a4485cd8 nodeName:}" failed. No retries permitted until 2026-07-10 13:38:18.784012107 +0000 UTC m=+1696.316614386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls") pod "maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" (UID: "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8") : secret "maas-api-e2e-mt-15039e79-serving-cert" not found Jul 10 13:38:18.284498 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.284480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.293330 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.293305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b46x\" (UniqueName: \"kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.788444 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.788413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.790795 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.790763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") pod \"maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:18.961267 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:18.961238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:19.099966 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:19.099942 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:38:19.102223 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:38:19.102191 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7132ca6b_28c2_453c_b4c9_0bd5a4485cd8.slice/crio-9c62423c4933efc6d139a904e894daef926ecc16167d17e6c3a0ae532442fa5c WatchSource:0}: Error finding container 9c62423c4933efc6d139a904e894daef926ecc16167d17e6c3a0ae532442fa5c: Status 404 returned error can't find the container with id 9c62423c4933efc6d139a904e894daef926ecc16167d17e6c3a0ae532442fa5c Jul 10 13:38:19.159046 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:19.159011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" event={"ID":"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8","Type":"ContainerStarted","Data":"9c62423c4933efc6d139a904e894daef926ecc16167d17e6c3a0ae532442fa5c"} Jul 10 13:38:20.164954 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:20.164922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" event={"ID":"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8","Type":"ContainerStarted","Data":"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2"} Jul 10 13:38:20.165400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:20.165065 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:20.183194 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:20.183149 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" podStartSLOduration=1.833303297 podStartE2EDuration="2.183134554s" totalImagesPullingTime="349.831257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:38:20.182238954 +0000 UTC m=+1697.714841265" watchObservedRunningTime="2026-07-10 13:38:20.183134554 +0000 UTC m=+1697.715736852" Jul 10 13:38:23.295604 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:23.295567 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:38:26.173958 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:26.173885 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:38:42.883978 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:42.883938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:38:57.488196 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:38:57.488152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:39:00.453138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.453101 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:39:00.453570 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.453370 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" podUID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" containerName="maas-api" containerID="cri-o://5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2" gracePeriod=30 Jul 10 13:39:00.697273 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.697249 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:39:00.741204 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.741124 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca\") pod \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " Jul 10 13:39:00.741332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.741222 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b46x\" (UniqueName: \"kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x\") pod \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " Jul 10 13:39:00.741332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.741317 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") pod \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\" (UID: \"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8\") " Jul 10 13:39:00.741497 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.741470 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" (UID: "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:39:00.741577 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.741559 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:39:00.743247 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.743213 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x" (OuterVolumeSpecName: "kube-api-access-4b46x") pod "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" (UID: "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8"). InnerVolumeSpecName "kube-api-access-4b46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:39:00.743352 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.743297 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" (UID: "7132ca6b-28c2-453c-b4c9-0bd5a4485cd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:39:00.842963 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.842942 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b46x\" (UniqueName: \"kubernetes.io/projected/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-kube-api-access-4b46x\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:39:00.842963 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:00.842964 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:39:01.364302 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.364267 2573 generic.go:356] "Generic (PLEG): container finished" podID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" containerID="5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2" exitCode=0 Jul 10 13:39:01.364484 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.364316 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" Jul 10 13:39:01.364484 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.364355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" event={"ID":"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8","Type":"ContainerDied","Data":"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2"} Jul 10 13:39:01.364484 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.364409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv" event={"ID":"7132ca6b-28c2-453c-b4c9-0bd5a4485cd8","Type":"ContainerDied","Data":"9c62423c4933efc6d139a904e894daef926ecc16167d17e6c3a0ae532442fa5c"} Jul 10 13:39:01.364484 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.364430 2573 scope.go:117] "RemoveContainer" containerID="5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2" Jul 10 13:39:01.373474 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.373457 2573 scope.go:117] "RemoveContainer" containerID="5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2" Jul 10 13:39:01.373726 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:39:01.373707 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2\": container with ID starting with 5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2 not found: ID does not exist" containerID="5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2" Jul 10 13:39:01.373782 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.373738 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2"} err="failed to get container status \"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2\": rpc error: code = NotFound desc = could not find container \"5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2\": container with ID starting with 5fe0c8eb4b0321700de7c96fd06c598bccbe3f42fc80e6366ff0e240ccd2e9c2 not found: ID does not exist" Jul 10 13:39:01.380472 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.380447 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:39:01.382160 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:01.382142 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15039e79-594dcd8c65-5xfdv"] Jul 10 13:39:02.970497 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:02.970465 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" path="/var/lib/kubelet/pods/7132ca6b-28c2-453c-b4c9-0bd5a4485cd8/volumes" Jul 10 13:39:10.887154 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:10.887118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:39:50.914165 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:39:50.914129 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:40:00.089695 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:00.089602 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:40:03.022648 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:03.022599 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:40:03.026322 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:03.026300 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:40:11.388552 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:11.388520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:40:52.030994 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.030959 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:40:52.031383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.031319 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" containerName="maas-api" Jul 10 13:40:52.031383 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.031329 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" containerName="maas-api" Jul 10 13:40:52.031500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.031399 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7132ca6b-28c2-453c-b4c9-0bd5a4485cd8" containerName="maas-api" Jul 10 13:40:52.034509 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.034492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.042327 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.042298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c7549e-serving-cert\"" Jul 10 13:40:52.042832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.042817 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:40:52.043465 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.043448 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:40:52.043544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.043453 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:40:52.049397 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.049368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.049568 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.049548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bns\" (UniqueName: \"kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.049664 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.049610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.055159 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.055134 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:40:52.150886 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.150856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.151042 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.150908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bns\" (UniqueName: \"kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.151042 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.150943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.151550 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.151533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.153199 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.153182 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.160198 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.160169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bns\" (UniqueName: \"kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns\") pod \"maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.345891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.345808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:52.485920 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.485897 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:40:52.488598 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:40:52.488568 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462ae004_0720_429b_9651_a27837fec3b5.slice/crio-521f6a9c769c2702080de29293ff934a4f5dacccf5845ceb2b3f75c1b51c7559 WatchSource:0}: Error finding container 521f6a9c769c2702080de29293ff934a4f5dacccf5845ceb2b3f75c1b51c7559: Status 404 returned error can't find the container with id 521f6a9c769c2702080de29293ff934a4f5dacccf5845ceb2b3f75c1b51c7559 Jul 10 13:40:52.489906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.489890 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:40:52.895305 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:52.895272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" event={"ID":"462ae004-0720-429b-9651-a27837fec3b5","Type":"ContainerStarted","Data":"521f6a9c769c2702080de29293ff934a4f5dacccf5845ceb2b3f75c1b51c7559"} Jul 10 13:40:53.904377 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:53.904341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" event={"ID":"462ae004-0720-429b-9651-a27837fec3b5","Type":"ContainerStarted","Data":"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532"} Jul 10 13:40:53.904789 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:53.904472 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:40:53.925650 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:53.925573 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" podStartSLOduration=1.350036937 podStartE2EDuration="1.925558951s" totalImagesPullingTime="575.522014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:40:53.924390272 +0000 UTC m=+1851.456992573" watchObservedRunningTime="2026-07-10 13:40:53.925558951 +0000 UTC m=+1851.458161250" Jul 10 13:40:59.915155 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:40:59.915125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:41:00.102563 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:41:00.102526 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:41:10.888390 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:41:10.888352 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:41:20.593028 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:41:20.592989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:41:47.785149 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:41:47.785106 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:42:00.775946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:00.775912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:42:46.998749 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:46.998720 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:42:46.999235 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:46.998910 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" podUID="462ae004-0720-429b-9651-a27837fec3b5" containerName="maas-api" containerID="cri-o://245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532" gracePeriod=30 Jul 10 13:42:47.256837 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.256779 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:42:47.323485 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.323458 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87bns\" (UniqueName: \"kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns\") pod \"462ae004-0720-429b-9651-a27837fec3b5\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " Jul 10 13:42:47.323671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.323509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca\") pod \"462ae004-0720-429b-9651-a27837fec3b5\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " Jul 10 13:42:47.323671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.323562 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls\") pod \"462ae004-0720-429b-9651-a27837fec3b5\" (UID: \"462ae004-0720-429b-9651-a27837fec3b5\") " Jul 10 13:42:47.323910 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.323885 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "462ae004-0720-429b-9651-a27837fec3b5" (UID: "462ae004-0720-429b-9651-a27837fec3b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:42:47.325651 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.325601 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns" (OuterVolumeSpecName: "kube-api-access-87bns") pod "462ae004-0720-429b-9651-a27837fec3b5" (UID: "462ae004-0720-429b-9651-a27837fec3b5"). InnerVolumeSpecName "kube-api-access-87bns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:42:47.325651 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.325613 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "462ae004-0720-429b-9651-a27837fec3b5" (UID: "462ae004-0720-429b-9651-a27837fec3b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:42:47.424528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.424501 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462ae004-0720-429b-9651-a27837fec3b5-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:42:47.424528 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.424527 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bns\" (UniqueName: \"kubernetes.io/projected/462ae004-0720-429b-9651-a27837fec3b5-kube-api-access-87bns\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:42:47.424728 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.424540 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462ae004-0720-429b-9651-a27837fec3b5-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:42:47.448600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.448573 2573 generic.go:356] "Generic (PLEG): container finished" podID="462ae004-0720-429b-9651-a27837fec3b5" containerID="245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532" exitCode=0 Jul 10 13:42:47.448742 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.448618 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" Jul 10 13:42:47.448742 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.448660 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" event={"ID":"462ae004-0720-429b-9651-a27837fec3b5","Type":"ContainerDied","Data":"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532"} Jul 10 13:42:47.448742 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.448720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s" event={"ID":"462ae004-0720-429b-9651-a27837fec3b5","Type":"ContainerDied","Data":"521f6a9c769c2702080de29293ff934a4f5dacccf5845ceb2b3f75c1b51c7559"} Jul 10 13:42:47.448868 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.448750 2573 scope.go:117] "RemoveContainer" containerID="245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532" Jul 10 13:42:47.457802 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.457781 2573 scope.go:117] "RemoveContainer" containerID="245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532" Jul 10 13:42:47.458077 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:42:47.458055 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532\": container with ID starting with 245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532 not found: ID does not exist" containerID="245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532" Jul 10 13:42:47.458154 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.458084 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532"} err="failed to get container status \"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532\": rpc error: code = NotFound desc = could not find container \"245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532\": container with ID starting with 245f3d674c99a6d7ff14dcdd3a3989118516f0d4721bdace9b6f0c029d507532 not found: ID does not exist" Jul 10 13:42:47.470188 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.470165 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:42:47.473707 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:47.473687 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c7549e-7c4c7f744f-dfd4s"] Jul 10 13:42:48.969068 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:48.969036 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462ae004-0720-429b-9651-a27837fec3b5" path="/var/lib/kubelet/pods/462ae004-0720-429b-9651-a27837fec3b5/volumes" Jul 10 13:42:49.586849 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:42:49.586819 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:43:32.983137 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:32.983109 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:43:37.395650 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.395608 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:43:37.396088 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.395988 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462ae004-0720-429b-9651-a27837fec3b5" containerName="maas-api" Jul 10 13:43:37.396088 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.396000 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="462ae004-0720-429b-9651-a27837fec3b5" containerName="maas-api" Jul 10 13:43:37.396088 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.396063 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="462ae004-0720-429b-9651-a27837fec3b5" containerName="maas-api" Jul 10 13:43:37.398941 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.398923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.401745 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.401717 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 13:43:37.401872 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.401765 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fec05d-openshift-default-dockercfg-cjh98\"" Jul 10 13:43:37.413059 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.413040 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:43:37.425303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425409 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425587 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425474 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425587 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425587 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.425734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.425617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxq9\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526501 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxq9\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526708 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526937 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526937 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.526937 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527097 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527097 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.526957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527097 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.527055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527253 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.527239 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527324 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.527305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.527818 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.527796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.529234 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.529213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.529465 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.529445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.534525 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.534503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxq9\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.534793 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.534774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token\") pod \"e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.710810 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.710732 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:37.849028 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:37.849005 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:43:37.851672 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:43:37.851642 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17fcc59_5238_4e0a_9d71_a9ea3fe78ec4.slice/crio-e0ee146af958e4e3e92f5e17edced897084606edcea3bca67f341c7e7af291df WatchSource:0}: Error finding container e0ee146af958e4e3e92f5e17edced897084606edcea3bca67f341c7e7af291df: Status 404 returned error can't find the container with id e0ee146af958e4e3e92f5e17edced897084606edcea3bca67f341c7e7af291df Jul 10 13:43:38.689915 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:38.689876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerStarted","Data":"e0ee146af958e4e3e92f5e17edced897084606edcea3bca67f341c7e7af291df"} Jul 10 13:43:40.407006 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:40.406966 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:40.407312 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:40.407039 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:40.407312 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:40.407068 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:40.701691 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:40.701580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerStarted","Data":"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b"} Jul 10 13:43:40.711380 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:40.711356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:41.715964 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:41.715934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:41.736079 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:41.736034 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podStartSLOduration=2.182969332 podStartE2EDuration="4.736017938s" totalImagesPullingTime="2.553048606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:43:40.751189303 +0000 UTC m=+2018.283791601" watchObservedRunningTime="2026-07-10 13:43:41.736017938 +0000 UTC m=+2019.268620237" Jul 10 13:43:42.713202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.713170 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:42.714095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.714078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:42.936385 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.936356 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:43:42.940053 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.940026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:42.942767 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.942746 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fec05d-serving-cert\"" Jul 10 13:43:42.942948 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.942827 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:43:42.943124 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.942895 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:43:42.943205 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.942897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:43:42.952519 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.952498 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:43:42.972072 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.972013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:42.972172 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.972119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:42.972172 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:42.972144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4f4v\" (UniqueName: \"kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.072599 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.072567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.072599 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.072604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f4v\" (UniqueName: \"kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.072832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.072664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.072832 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:43:43.072736 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-serving-cert: secret "maas-api-e2e-api-a-fec05d-serving-cert" not found Jul 10 13:43:43.072832 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:43:43.072807 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls podName:45f5fb18-09d2-4e74-a18a-13325ef1d25c nodeName:}" failed. No retries permitted until 2026-07-10 13:43:43.572780472 +0000 UTC m=+2021.105382749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls") pod "maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" (UID: "45f5fb18-09d2-4e74-a18a-13325ef1d25c") : secret "maas-api-e2e-api-a-fec05d-serving-cert" not found Jul 10 13:43:43.073390 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.073372 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.090345 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.090311 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f4v\" (UniqueName: \"kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.576764 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.576714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.579202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.579181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") pod \"maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:43.852449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:43.852368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:44.014374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:44.014346 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:43:44.016401 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:43:44.016374 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f5fb18_09d2_4e74_a18a_13325ef1d25c.slice/crio-07b403c94ca254b5a32cdf31eb1009a0648af3f6f93794e695be1c79b4a109af WatchSource:0}: Error finding container 07b403c94ca254b5a32cdf31eb1009a0648af3f6f93794e695be1c79b4a109af: Status 404 returned error can't find the container with id 07b403c94ca254b5a32cdf31eb1009a0648af3f6f93794e695be1c79b4a109af Jul 10 13:43:44.724616 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:44.724526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" event={"ID":"45f5fb18-09d2-4e74-a18a-13325ef1d25c","Type":"ContainerStarted","Data":"e85efe555348989e408eb3f51af908ac4c487b0ef855e0180ae55e1a05361185"} Jul 10 13:43:44.724616 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:44.724587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" event={"ID":"45f5fb18-09d2-4e74-a18a-13325ef1d25c","Type":"ContainerStarted","Data":"07b403c94ca254b5a32cdf31eb1009a0648af3f6f93794e695be1c79b4a109af"} Jul 10 13:43:44.724901 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:44.724618 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:44.741141 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:44.741059 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" podStartSLOduration=2.288044863 podStartE2EDuration="2.741042183s" totalImagesPullingTime="452.99732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:43:44.739177772 +0000 UTC m=+2022.271780071" watchObservedRunningTime="2026-07-10 13:43:44.741042183 +0000 UTC m=+2022.273644483" Jul 10 13:43:46.288254 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:46.288225 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:43:50.751125 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:50.751094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:43:54.785666 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.785619 2573 generic.go:356] "Generic (PLEG): container finished" podID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerID="242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b" exitCode=0 Jul 10 13:43:54.786074 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.785694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerDied","Data":"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b"} Jul 10 13:43:54.786074 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.786054 2573 scope.go:117] "RemoveContainer" containerID="242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b" Jul 10 13:43:54.787941 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.787916 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:54.788010 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.787984 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:54.788054 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:54.788016 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:43:55.792747 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:55.792717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerStarted","Data":"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7"} Jul 10 13:43:56.711587 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:56.711551 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:56.712618 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:56.712591 2573 patch_prober.go:28] interesting pod/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 13:43:56.712746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:56.712664 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 13:43:57.711806 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:57.711769 2573 patch_prober.go:28] interesting pod/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 13:43:57.712193 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:57.711838 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 13:43:57.713907 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:57.713888 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:58.711066 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:58.711030 2573 patch_prober.go:28] interesting pod/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 13:43:58.711237 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:58.711081 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 13:43:59.728599 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:59.728569 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:43:59.729555 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:43:59.729535 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:44:04.092864 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.092829 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:44:04.615057 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.615022 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:44:04.615283 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.615259 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" podUID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" containerName="maas-api" containerID="cri-o://e85efe555348989e408eb3f51af908ac4c487b0ef855e0180ae55e1a05361185" gracePeriod=30 Jul 10 13:44:04.838182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.838152 2573 generic.go:356] "Generic (PLEG): container finished" podID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" containerID="e85efe555348989e408eb3f51af908ac4c487b0ef855e0180ae55e1a05361185" exitCode=0 Jul 10 13:44:04.838353 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.838225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" event={"ID":"45f5fb18-09d2-4e74-a18a-13325ef1d25c","Type":"ContainerDied","Data":"e85efe555348989e408eb3f51af908ac4c487b0ef855e0180ae55e1a05361185"} Jul 10 13:44:04.873312 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.873284 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:44:04.963980 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.963953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") pod \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " Jul 10 13:44:04.964140 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.964028 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca\") pod \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " Jul 10 13:44:04.964140 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.964068 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4f4v\" (UniqueName: \"kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v\") pod \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\" (UID: \"45f5fb18-09d2-4e74-a18a-13325ef1d25c\") " Jul 10 13:44:04.964385 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.964356 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45f5fb18-09d2-4e74-a18a-13325ef1d25c" (UID: "45f5fb18-09d2-4e74-a18a-13325ef1d25c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:44:04.966107 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.966073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45f5fb18-09d2-4e74-a18a-13325ef1d25c" (UID: "45f5fb18-09d2-4e74-a18a-13325ef1d25c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:44:04.966220 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:04.966180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v" (OuterVolumeSpecName: "kube-api-access-z4f4v") pod "45f5fb18-09d2-4e74-a18a-13325ef1d25c" (UID: "45f5fb18-09d2-4e74-a18a-13325ef1d25c"). InnerVolumeSpecName "kube-api-access-z4f4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:44:05.065605 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.065573 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f5fb18-09d2-4e74-a18a-13325ef1d25c-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:05.065605 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.065596 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4f4v\" (UniqueName: \"kubernetes.io/projected/45f5fb18-09d2-4e74-a18a-13325ef1d25c-kube-api-access-z4f4v\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:05.065605 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.065607 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f5fb18-09d2-4e74-a18a-13325ef1d25c-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:05.844653 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.844608 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" Jul 10 13:44:05.844653 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.844624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t" event={"ID":"45f5fb18-09d2-4e74-a18a-13325ef1d25c","Type":"ContainerDied","Data":"07b403c94ca254b5a32cdf31eb1009a0648af3f6f93794e695be1c79b4a109af"} Jul 10 13:44:05.845097 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.844702 2573 scope.go:117] "RemoveContainer" containerID="e85efe555348989e408eb3f51af908ac4c487b0ef855e0180ae55e1a05361185" Jul 10 13:44:05.863595 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.863574 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:44:05.866050 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:05.866031 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fec05d-6f9fc7dc6-zbl6t"] Jul 10 13:44:06.968895 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:06.968857 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" path="/var/lib/kubelet/pods/45f5fb18-09d2-4e74-a18a-13325ef1d25c/volumes" Jul 10 13:44:10.443761 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:10.443729 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:44:10.444153 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:10.443941 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" containerID="cri-o://3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7" gracePeriod=30 Jul 10 13:44:14.731559 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:14.731522 2573 patch_prober.go:28] interesting pod/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 13:44:14.732037 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:14.731595 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:44:14.784820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:14.784788 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:44:15.699483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.699463 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:44:15.858249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858174 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858212 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khxq9\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858249 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858232 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858259 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858284 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858309 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858367 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858414 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs\") pod \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\" (UID: \"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4\") " Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858501 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858667 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.858820 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858740 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:15.859297 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.858977 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:15.859297 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.859045 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:44:15.859297 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.859271 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:15.860541 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.860518 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:44:15.860675 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.860541 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9" (OuterVolumeSpecName: "kube-api-access-khxq9") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "kube-api-access-khxq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:44:15.860675 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.860568 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:15.860763 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.860742 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" (UID: "d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:44:15.899191 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.899164 2573 generic.go:356] "Generic (PLEG): container finished" podID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerID="3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7" exitCode=0 Jul 10 13:44:15.899310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.899215 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" Jul 10 13:44:15.899310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.899219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerDied","Data":"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7"} Jul 10 13:44:15.899310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.899262 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr" event={"ID":"d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4","Type":"ContainerDied","Data":"e0ee146af958e4e3e92f5e17edced897084606edcea3bca67f341c7e7af291df"} Jul 10 13:44:15.899310 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.899283 2573 scope.go:117] "RemoveContainer" containerID="3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7" Jul 10 13:44:15.908123 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.908104 2573 scope.go:117] "RemoveContainer" containerID="242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b" Jul 10 13:44:15.915838 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.915823 2573 scope.go:117] "RemoveContainer" containerID="3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7" Jul 10 13:44:15.916074 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:44:15.916055 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7\": container with ID starting with 3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7 not found: ID does not exist" containerID="3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7" Jul 10 13:44:15.916131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.916080 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7"} err="failed to get container status \"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7\": rpc error: code = NotFound desc = could not find container \"3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7\": container with ID starting with 3a5e1ee9fc813d9c9180054277ad86daec2cd5a5f862c6372dff7e27e84acbe7 not found: ID does not exist" Jul 10 13:44:15.916131 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.916098 2573 scope.go:117] "RemoveContainer" containerID="242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b" Jul 10 13:44:15.916313 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:44:15.916298 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b\": container with ID starting with 242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b not found: ID does not exist" containerID="242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b" Jul 10 13:44:15.916358 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.916318 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b"} err="failed to get container status \"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b\": rpc error: code = NotFound desc = could not find container \"242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b\": container with ID starting with 242d185f5d64fb6286c18e76ddde9438e8071fb58612840429a61f7eec7d278b not found: ID does not exist" Jul 10 13:44:15.925186 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.925166 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:44:15.930138 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.930118 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fec05d-openshift-default-596dbf68b9-rkwzr"] Jul 10 13:44:15.959927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959904 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-podinfo\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.959927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959928 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-token\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959939 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-credential-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959949 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-envoy\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959957 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istiod-ca-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959965 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-workload-certs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959973 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khxq9\" (UniqueName: \"kubernetes.io/projected/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-kube-api-access-khxq9\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:15.960062 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:15.959980 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4-istio-data\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:44:16.969537 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:16.969502 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" path="/var/lib/kubelet/pods/d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4/volumes" Jul 10 13:44:18.533464 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.533432 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:44:18.534014 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.533996 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:18.534063 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534017 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:18.534063 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534054 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" containerName="maas-api" Jul 10 13:44:18.534128 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534064 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" containerName="maas-api" Jul 10 13:44:18.534161 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534140 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:18.534161 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534155 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:18.534222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.534166 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f5fb18-09d2-4e74-a18a-13325ef1d25c" containerName="maas-api" Jul 10 13:44:18.539019 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.538998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.541411 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.541391 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 13:44:18.541538 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.541395 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-90a164-openshift-default-dockercfg-kr87c\"" Jul 10 13:44:18.547737 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.547713 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:44:18.680854 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.680823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.680868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.680935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.680960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.681010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681048 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.681034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681289 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.681060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681289 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.681101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.681289 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.681148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqzm\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.781923 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.781893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.781932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.781953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.781990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782073 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782390 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqzm\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782390 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782497 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782470 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782579 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782548 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782827 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.782888 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.782827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.784320 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.784261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.784693 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.784675 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.790239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.790215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.790450 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.790431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqzm\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm\") pod \"e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:18.851334 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:18.851307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:19.009526 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.009498 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:44:19.012327 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:44:19.012291 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded85cd47_6028_4656_aafb_6ff99af1e1a2.slice/crio-d4d9adb3f50a933738e324b5b878d063deafb26ca8dde9b534e9684fa74e35ba WatchSource:0}: Error finding container d4d9adb3f50a933738e324b5b878d063deafb26ca8dde9b534e9684fa74e35ba: Status 404 returned error can't find the container with id d4d9adb3f50a933738e324b5b878d063deafb26ca8dde9b534e9684fa74e35ba Jul 10 13:44:19.014548 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.014510 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:19.014619 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.014604 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:19.014686 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.014665 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:19.926338 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.926302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerStarted","Data":"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2"} Jul 10 13:44:19.926750 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.926365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerStarted","Data":"d4d9adb3f50a933738e324b5b878d063deafb26ca8dde9b534e9684fa74e35ba"} Jul 10 13:44:19.949087 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:19.949041 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podStartSLOduration=1.949026275 podStartE2EDuration="1.949026275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:44:19.946780907 +0000 UTC m=+2057.479383216" watchObservedRunningTime="2026-07-10 13:44:19.949026275 +0000 UTC m=+2057.481628574" Jul 10 13:44:20.852265 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:20.852233 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:20.856822 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:20.856798 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:20.931219 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:20.931186 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:20.932164 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:20.932145 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:21.972848 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.972808 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:44:21.973357 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.973342 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:21.973404 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.973360 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17fcc59-5238-4e0a-9d71-a9ea3fe78ec4" containerName="istio-proxy" Jul 10 13:44:21.976772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.976754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:21.979448 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.979422 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:44:21.979576 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.979459 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-90a164-serving-cert\"" Jul 10 13:44:21.980267 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.980248 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz4r\"" Jul 10 13:44:21.980355 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.980254 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:44:21.987923 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:21.987900 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:44:22.109521 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.109485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.109746 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.109730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jv6\" (UniqueName: \"kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.109816 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.109766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.210456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.210428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.210621 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.210511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.210621 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.210600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jv6\" (UniqueName: \"kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.211211 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.211189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.212939 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.212914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.219037 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.219012 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jv6\" (UniqueName: \"kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6\") pod \"maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.289524 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.289458 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:22.430576 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.430554 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:44:22.433089 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:44:22.433060 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9775a5a2_0f78_4e45_b78f_09f3eadfc33c.slice/crio-49046fe05026e77f43aff8a224899d24e9d75d0df5e762ae35f3fb37218c8292 WatchSource:0}: Error finding container 49046fe05026e77f43aff8a224899d24e9d75d0df5e762ae35f3fb37218c8292: Status 404 returned error can't find the container with id 49046fe05026e77f43aff8a224899d24e9d75d0df5e762ae35f3fb37218c8292 Jul 10 13:44:22.942921 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.942886 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" event={"ID":"9775a5a2-0f78-4e45-b78f-09f3eadfc33c","Type":"ContainerStarted","Data":"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f"} Jul 10 13:44:22.943090 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.942937 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" event={"ID":"9775a5a2-0f78-4e45-b78f-09f3eadfc33c","Type":"ContainerStarted","Data":"49046fe05026e77f43aff8a224899d24e9d75d0df5e762ae35f3fb37218c8292"} Jul 10 13:44:22.960462 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:22.960416 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" podStartSLOduration=1.556213386 podStartE2EDuration="1.960402042s" totalImagesPullingTime="404.188656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:44:22.959284564 +0000 UTC m=+2060.491886878" watchObservedRunningTime="2026-07-10 13:44:22.960402042 +0000 UTC m=+2060.493004342" Jul 10 13:44:23.947398 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:23.947367 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:27.057227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.057152 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:44:27.060803 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.060786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.063747 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.063724 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-efd2ad-openshift-default-dockercfg-2fsbw\"" Jul 10 13:44:27.078302 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.078279 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:44:27.155498 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155498 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155724 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ks5\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.155916 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.155743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.256786 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.256752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ks5\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.256786 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.256788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257003 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.256854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257003 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.256882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257111 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257241 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257732 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257422 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257732 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257732 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257675 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257901 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.257901 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.257840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.259315 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.259288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.259669 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.259646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.266094 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.266072 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.266447 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.266427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ks5\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5\") pod \"e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.371821 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.371731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:27.516533 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.516512 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:44:27.518603 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:44:27.518570 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a9a067_2338_43d6_b8ea_963d49bcb547.slice/crio-ed1eb3b33eb636e7457e0eb05926da3d756bc6915558b0d657d9aaed1596a077 WatchSource:0}: Error finding container ed1eb3b33eb636e7457e0eb05926da3d756bc6915558b0d657d9aaed1596a077: Status 404 returned error can't find the container with id ed1eb3b33eb636e7457e0eb05926da3d756bc6915558b0d657d9aaed1596a077 Jul 10 13:44:27.520711 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.520682 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:27.520806 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.520743 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:27.520806 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.520775 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:27.969195 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.969156 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerStarted","Data":"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7"} Jul 10 13:44:27.969347 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.969217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerStarted","Data":"ed1eb3b33eb636e7457e0eb05926da3d756bc6915558b0d657d9aaed1596a077"} Jul 10 13:44:27.992034 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:27.991982 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podStartSLOduration=0.991966359 podStartE2EDuration="991.966359ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:44:27.989390445 +0000 UTC m=+2065.521992744" watchObservedRunningTime="2026-07-10 13:44:27.991966359 +0000 UTC m=+2065.524568730" Jul 10 13:44:28.372034 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:28.371963 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:28.376837 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:28.376811 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:28.975801 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:28.975771 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:28.976832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:28.976811 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:29.384316 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:29.384282 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:44:29.958422 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:29.958393 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:44:36.972803 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:36.972765 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 13:44:36.994490 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:36.972842 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 13:44:38.022801 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.022767 2573 generic.go:356] "Generic (PLEG): container finished" podID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerID="eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2" exitCode=0 Jul 10 13:44:38.023172 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.022839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerDied","Data":"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2"} Jul 10 13:44:38.023237 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.023221 2573 scope.go:117] "RemoveContainer" containerID="eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2" Jul 10 13:44:38.024257 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.024228 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:38.024328 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.024299 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:38.024376 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:38.024330 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:39.041444 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:39.041408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerStarted","Data":"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e"} Jul 10 13:44:39.852263 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:39.852222 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:39.852482 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:39.852361 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 13:44:39.852482 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:39.852408 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 13:44:40.852577 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:40.852532 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 13:44:40.853081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:40.852599 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 13:44:41.852582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:41.852534 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 13:44:41.853068 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:41.852607 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 13:44:42.852187 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:42.852155 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 13:44:42.852387 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:42.852206 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 13:44:43.856755 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:43.856724 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:43.857191 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:43.857000 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:43.857728 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:43.857708 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:44:43.986388 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:43.986356 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:44:45.016787 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:45.016751 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 13:44:45.017171 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:45.016820 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 13:44:46.089801 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.089767 2573 generic.go:356] "Generic (PLEG): container finished" podID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerID="3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7" exitCode=0 Jul 10 13:44:46.090222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.089844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerDied","Data":"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7"} Jul 10 13:44:46.090265 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.090236 2573 scope.go:117] "RemoveContainer" containerID="3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7" Jul 10 13:44:46.091267 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.091242 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:46.091377 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.091304 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:46.091377 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:46.091335 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 13:44:47.097144 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:47.097109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerStarted","Data":"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099"} Jul 10 13:44:47.372365 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:47.372343 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:47.372625 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:47.372601 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 13:44:47.372764 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:47.372672 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 13:44:48.372315 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:48.372282 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 13:44:48.372739 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:48.372336 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 13:44:49.372344 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:49.372304 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 13:44:49.372855 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:49.372358 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 13:44:50.372928 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:50.372897 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 13:44:50.373391 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:50.373363 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 13:44:51.375191 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:51.375160 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:51.375627 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:51.375600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:44:51.376295 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:44:51.376278 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:45:03.055222 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:45:03.055194 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:45:03.059394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:45:03.059374 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:45:05.377154 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:45:05.377122 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:45:35.186981 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:45:35.186949 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:45:56.895971 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:45:56.895886 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:46:15.922044 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:46:15.922013 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:46:32.493753 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:46:32.493714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:46:51.286608 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:46:51.286572 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:47:09.276728 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:47:09.276688 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:47:27.676539 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:47:27.676458 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:47:42.590804 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:47:42.590771 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:48:07.186952 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:48:07.186918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:48:45.890453 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:48:45.890416 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:49:08.294433 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:49:08.294344 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:49:27.776521 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:49:27.776490 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:49:40.390104 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:49:40.390066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:50:03.085229 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:50:03.085203 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:50:03.091198 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:50:03.091176 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:50:07.389140 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:50:07.389110 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:50:46.580948 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:50:46.580915 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:51:09.187975 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:09.187941 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:51:14.773783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:14.773753 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156710692272"] Jul 10 13:51:14.777296 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:14.777280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:14.779535 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:14.779514 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-q786g\"" Jul 10 13:51:14.782501 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:14.782473 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156710692272"] Jul 10 13:51:14.967135 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:14.967110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rvs\" (UniqueName: \"kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs\") pod \"test-curl-7212-140156710692272\" (UID: \"a9b70fc1-098a-42a5-a413-77118b2568d8\") " pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:15.068601 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:15.068514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rvs\" (UniqueName: \"kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs\") pod \"test-curl-7212-140156710692272\" (UID: \"a9b70fc1-098a-42a5-a413-77118b2568d8\") " pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:15.078014 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:15.077991 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rvs\" (UniqueName: \"kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs\") pod \"test-curl-7212-140156710692272\" (UID: \"a9b70fc1-098a-42a5-a413-77118b2568d8\") " pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:15.086850 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:15.086825 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:15.225489 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:15.221940 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156710692272"] Jul 10 13:51:15.227778 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:15.227747 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b70fc1_098a_42a5_a413_77118b2568d8.slice/crio-421a732b923d3ffb13347bbaad0f532b3141c1605a25e27bb67508103de6bad0 WatchSource:0}: Error finding container 421a732b923d3ffb13347bbaad0f532b3141c1605a25e27bb67508103de6bad0: Status 404 returned error can't find the container with id 421a732b923d3ffb13347bbaad0f532b3141c1605a25e27bb67508103de6bad0 Jul 10 13:51:15.229790 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:15.229769 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:51:16.030492 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:16.030461 2573 generic.go:356] "Generic (PLEG): container finished" podID="a9b70fc1-098a-42a5-a413-77118b2568d8" containerID="ff1f22fbdba0b10dd10a0c3dc3a221702b3526288dd9d1d4cc37a39598f59e09" exitCode=0 Jul 10 13:51:16.030826 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:16.030512 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156710692272" event={"ID":"a9b70fc1-098a-42a5-a413-77118b2568d8","Type":"ContainerDied","Data":"ff1f22fbdba0b10dd10a0c3dc3a221702b3526288dd9d1d4cc37a39598f59e09"} Jul 10 13:51:16.030826 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:16.030545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156710692272" event={"ID":"a9b70fc1-098a-42a5-a413-77118b2568d8","Type":"ContainerStarted","Data":"421a732b923d3ffb13347bbaad0f532b3141c1605a25e27bb67508103de6bad0"} Jul 10 13:51:17.177133 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.177112 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:17.188178 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.188154 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5rvs\" (UniqueName: \"kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs\") pod \"a9b70fc1-098a-42a5-a413-77118b2568d8\" (UID: \"a9b70fc1-098a-42a5-a413-77118b2568d8\") " Jul 10 13:51:17.190057 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.190036 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs" (OuterVolumeSpecName: "kube-api-access-s5rvs") pod "a9b70fc1-098a-42a5-a413-77118b2568d8" (UID: "a9b70fc1-098a-42a5-a413-77118b2568d8"). InnerVolumeSpecName "kube-api-access-s5rvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:17.215582 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.215563 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156710692272_a9b70fc1-098a-42a5-a413-77118b2568d8/test-curl-7212-140156710692272/0.log" Jul 10 13:51:17.240278 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.240254 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156710692272"] Jul 10 13:51:17.243550 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.243529 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156710692272"] Jul 10 13:51:17.288808 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.288757 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rvs\" (UniqueName: \"kubernetes.io/projected/a9b70fc1-098a-42a5-a413-77118b2568d8-kube-api-access-s5rvs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:17.371374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.371347 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156710696880"] Jul 10 13:51:17.371865 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.371839 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b70fc1-098a-42a5-a413-77118b2568d8" containerName="test-curl-7212-140156710692272" Jul 10 13:51:17.371865 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.371860 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b70fc1-098a-42a5-a413-77118b2568d8" containerName="test-curl-7212-140156710692272" Jul 10 13:51:17.372022 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.371981 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b70fc1-098a-42a5-a413-77118b2568d8" containerName="test-curl-7212-140156710692272" Jul 10 13:51:17.375065 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.375046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:17.381445 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.381423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156710696880"] Jul 10 13:51:17.389349 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.389249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2ch\" (UniqueName: \"kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch\") pod \"test-curl-7212-140156710696880\" (UID: \"35f3b533-a00b-44b5-a8c1-c9d496feb5a3\") " pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:17.489720 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.489691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2ch\" (UniqueName: \"kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch\") pod \"test-curl-7212-140156710696880\" (UID: \"35f3b533-a00b-44b5-a8c1-c9d496feb5a3\") " pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:17.505011 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.504985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2ch\" (UniqueName: \"kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch\") pod \"test-curl-7212-140156710696880\" (UID: \"35f3b533-a00b-44b5-a8c1-c9d496feb5a3\") " pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:17.685400 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.685370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:17.825617 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:17.825592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156710696880"] Jul 10 13:51:17.827828 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:17.827797 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f3b533_a00b_44b5_a8c1_c9d496feb5a3.slice/crio-cbb99e227e62bb304d00f05f673937a590e7ff4535e49270106467fd44a028a0 WatchSource:0}: Error finding container cbb99e227e62bb304d00f05f673937a590e7ff4535e49270106467fd44a028a0: Status 404 returned error can't find the container with id cbb99e227e62bb304d00f05f673937a590e7ff4535e49270106467fd44a028a0 Jul 10 13:51:18.044130 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:18.044047 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156710696880" event={"ID":"35f3b533-a00b-44b5-a8c1-c9d496feb5a3","Type":"ContainerStarted","Data":"cbb99e227e62bb304d00f05f673937a590e7ff4535e49270106467fd44a028a0"} Jul 10 13:51:18.045251 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:18.045229 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:18.045373 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:18.045253 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421a732b923d3ffb13347bbaad0f532b3141c1605a25e27bb67508103de6bad0" Jul 10 13:51:18.055067 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:18.055043 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7212-140156710692272\" is forbidden: User \"system:node:ip-10-0-128-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-249.ec2.internal' and this object" podUID="a9b70fc1-098a-42a5-a413-77118b2568d8" pod="openshift-ingress/test-curl-7212-140156710692272" Jul 10 13:51:18.972804 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:18.972767 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b70fc1-098a-42a5-a413-77118b2568d8" path="/var/lib/kubelet/pods/a9b70fc1-098a-42a5-a413-77118b2568d8/volumes" Jul 10 13:51:19.051889 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:19.051854 2573 generic.go:356] "Generic (PLEG): container finished" podID="35f3b533-a00b-44b5-a8c1-c9d496feb5a3" containerID="84737f4f255a6edf6b8c87aac5637851abd930b693aa1ab33e9685ac55da625e" exitCode=0 Jul 10 13:51:19.052324 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:19.051899 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156710696880" event={"ID":"35f3b533-a00b-44b5-a8c1-c9d496feb5a3","Type":"ContainerDied","Data":"84737f4f255a6edf6b8c87aac5637851abd930b693aa1ab33e9685ac55da625e"} Jul 10 13:51:20.192600 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.192579 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:20.208573 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.208550 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s2ch\" (UniqueName: \"kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch\") pod \"35f3b533-a00b-44b5-a8c1-c9d496feb5a3\" (UID: \"35f3b533-a00b-44b5-a8c1-c9d496feb5a3\") " Jul 10 13:51:20.210624 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.210597 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch" (OuterVolumeSpecName: "kube-api-access-7s2ch") pod "35f3b533-a00b-44b5-a8c1-c9d496feb5a3" (UID: "35f3b533-a00b-44b5-a8c1-c9d496feb5a3"). InnerVolumeSpecName "kube-api-access-7s2ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:20.211938 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.211919 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156710696880_35f3b533-a00b-44b5-a8c1-c9d496feb5a3/test-curl-7212-140156710696880/0.log" Jul 10 13:51:20.235878 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.235853 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156710696880"] Jul 10 13:51:20.239760 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.239738 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156710696880"] Jul 10 13:51:20.309285 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.309210 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2ch\" (UniqueName: \"kubernetes.io/projected/35f3b533-a00b-44b5-a8c1-c9d496feb5a3-kube-api-access-7s2ch\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:20.393618 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.393575 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:20.394149 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.394130 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f3b533-a00b-44b5-a8c1-c9d496feb5a3" containerName="test-curl-7212-140156710696880" Jul 10 13:51:20.394200 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.394152 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f3b533-a00b-44b5-a8c1-c9d496feb5a3" containerName="test-curl-7212-140156710696880" Jul 10 13:51:20.394239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.394234 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f3b533-a00b-44b5-a8c1-c9d496feb5a3" containerName="test-curl-7212-140156710696880" Jul 10 13:51:20.397313 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.397296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:20.401085 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.401058 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:20.410158 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.410034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzt94\" (UniqueName: \"kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94\") pod \"test-curl-7212-140156711137904\" (UID: \"abf22496-b610-490e-8c26-d60dd1b7dd72\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:20.510514 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.510487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzt94\" (UniqueName: \"kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94\") pod \"test-curl-7212-140156711137904\" (UID: \"abf22496-b610-490e-8c26-d60dd1b7dd72\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:20.518668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.518646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzt94\" (UniqueName: \"kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94\") pod \"test-curl-7212-140156711137904\" (UID: \"abf22496-b610-490e-8c26-d60dd1b7dd72\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:20.707174 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.707132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:20.835216 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.835187 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:20.838565 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:20.838540 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf22496_b610_490e_8c26_d60dd1b7dd72.slice/crio-1eff7fa72622a4fb8be38502681a398ea432dd7ec13f90c870f15a2079e44aaa WatchSource:0}: Error finding container 1eff7fa72622a4fb8be38502681a398ea432dd7ec13f90c870f15a2079e44aaa: Status 404 returned error can't find the container with id 1eff7fa72622a4fb8be38502681a398ea432dd7ec13f90c870f15a2079e44aaa Jul 10 13:51:20.970028 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:20.969950 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f3b533-a00b-44b5-a8c1-c9d496feb5a3" path="/var/lib/kubelet/pods/35f3b533-a00b-44b5-a8c1-c9d496feb5a3/volumes" Jul 10 13:51:21.062771 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:21.062743 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156710696880" Jul 10 13:51:21.062771 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:21.062762 2573 scope.go:117] "RemoveContainer" containerID="84737f4f255a6edf6b8c87aac5637851abd930b693aa1ab33e9685ac55da625e" Jul 10 13:51:21.064095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:21.064070 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711137904" event={"ID":"abf22496-b610-490e-8c26-d60dd1b7dd72","Type":"ContainerStarted","Data":"1eff7fa72622a4fb8be38502681a398ea432dd7ec13f90c870f15a2079e44aaa"} Jul 10 13:51:22.070280 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:22.070249 2573 generic.go:356] "Generic (PLEG): container finished" podID="abf22496-b610-490e-8c26-d60dd1b7dd72" containerID="f966756f86639852e87f882630def48cc15bf3627bc68e2127c70a4ce70724fc" exitCode=0 Jul 10 13:51:22.070655 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:22.070331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711137904" event={"ID":"abf22496-b610-490e-8c26-d60dd1b7dd72","Type":"ContainerDied","Data":"f966756f86639852e87f882630def48cc15bf3627bc68e2127c70a4ce70724fc"} Jul 10 13:51:23.233964 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.233934 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.252783 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.252756 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156711137904_abf22496-b610-490e-8c26-d60dd1b7dd72/test-curl-7212-140156711137904/0.log" Jul 10 13:51:23.273689 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.273658 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:23.277909 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.277887 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:23.331446 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.331343 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzt94\" (UniqueName: \"kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94\") pod \"abf22496-b610-490e-8c26-d60dd1b7dd72\" (UID: \"abf22496-b610-490e-8c26-d60dd1b7dd72\") " Jul 10 13:51:23.333507 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.333470 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94" (OuterVolumeSpecName: "kube-api-access-nzt94") pod "abf22496-b610-490e-8c26-d60dd1b7dd72" (UID: "abf22496-b610-490e-8c26-d60dd1b7dd72"). InnerVolumeSpecName "kube-api-access-nzt94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:23.397132 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.397105 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:23.397527 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.397513 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf22496-b610-490e-8c26-d60dd1b7dd72" containerName="test-curl-7212-140156711137904" Jul 10 13:51:23.397577 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.397529 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf22496-b610-490e-8c26-d60dd1b7dd72" containerName="test-curl-7212-140156711137904" Jul 10 13:51:23.397613 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.397588 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf22496-b610-490e-8c26-d60dd1b7dd72" containerName="test-curl-7212-140156711137904" Jul 10 13:51:23.400612 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.400587 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:23.400725 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.400697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.432300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.432280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz68\" (UniqueName: \"kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68\") pod \"test-curl-7212-140156711137904\" (UID: \"f3f936c5-1a99-4646-9afd-46c96d53bf6b\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.432440 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.432423 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzt94\" (UniqueName: \"kubernetes.io/projected/abf22496-b610-490e-8c26-d60dd1b7dd72-kube-api-access-nzt94\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:23.533378 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.533341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz68\" (UniqueName: \"kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68\") pod \"test-curl-7212-140156711137904\" (UID: \"f3f936c5-1a99-4646-9afd-46c96d53bf6b\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.541202 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.541165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz68\" (UniqueName: \"kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68\") pod \"test-curl-7212-140156711137904\" (UID: \"f3f936c5-1a99-4646-9afd-46c96d53bf6b\") " pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.710710 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.710675 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:23.847984 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:23.847960 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:23.849799 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:23.849773 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3f936c5_1a99_4646_9afd_46c96d53bf6b.slice/crio-8fa5c77aa9c3d861c01cf7fecc1bced7ea6540bf5a913957fb9af61b7de3bc8f WatchSource:0}: Error finding container 8fa5c77aa9c3d861c01cf7fecc1bced7ea6540bf5a913957fb9af61b7de3bc8f: Status 404 returned error can't find the container with id 8fa5c77aa9c3d861c01cf7fecc1bced7ea6540bf5a913957fb9af61b7de3bc8f Jul 10 13:51:24.083255 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:24.083173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711137904" event={"ID":"f3f936c5-1a99-4646-9afd-46c96d53bf6b","Type":"ContainerStarted","Data":"8fa5c77aa9c3d861c01cf7fecc1bced7ea6540bf5a913957fb9af61b7de3bc8f"} Jul 10 13:51:24.084386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:24.084368 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:24.084499 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:24.084367 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eff7fa72622a4fb8be38502681a398ea432dd7ec13f90c870f15a2079e44aaa" Jul 10 13:51:24.095417 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:24.095390 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7212-140156711137904" oldPodUID="abf22496-b610-490e-8c26-d60dd1b7dd72" podUID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" Jul 10 13:51:24.969555 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:24.969520 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf22496-b610-490e-8c26-d60dd1b7dd72" path="/var/lib/kubelet/pods/abf22496-b610-490e-8c26-d60dd1b7dd72/volumes" Jul 10 13:51:25.090906 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:25.090874 2573 generic.go:356] "Generic (PLEG): container finished" podID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" containerID="16ee5888b90c1306f48647bd334abfce130fb5b7a6c8bdecdbb8d4218bf75a78" exitCode=0 Jul 10 13:51:25.091078 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:25.090954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711137904" event={"ID":"f3f936c5-1a99-4646-9afd-46c96d53bf6b","Type":"ContainerDied","Data":"16ee5888b90c1306f48647bd334abfce130fb5b7a6c8bdecdbb8d4218bf75a78"} Jul 10 13:51:26.242020 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.241994 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:26.255680 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.255653 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttz68\" (UniqueName: \"kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68\") pod \"f3f936c5-1a99-4646-9afd-46c96d53bf6b\" (UID: \"f3f936c5-1a99-4646-9afd-46c96d53bf6b\") " Jul 10 13:51:26.257939 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.257915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68" (OuterVolumeSpecName: "kube-api-access-ttz68") pod "f3f936c5-1a99-4646-9afd-46c96d53bf6b" (UID: "f3f936c5-1a99-4646-9afd-46c96d53bf6b"). InnerVolumeSpecName "kube-api-access-ttz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:26.262308 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.262282 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156711137904_f3f936c5-1a99-4646-9afd-46c96d53bf6b/test-curl-7212-140156711137904/0.log" Jul 10 13:51:26.282841 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.282813 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:26.286125 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.286105 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156711137904"] Jul 10 13:51:26.356352 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.356292 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttz68\" (UniqueName: \"kubernetes.io/projected/f3f936c5-1a99-4646-9afd-46c96d53bf6b-kube-api-access-ttz68\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:26.405904 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.405880 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:26.406267 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.406256 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" containerName="test-curl-7212-140156711137904" Jul 10 13:51:26.406306 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.406269 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" containerName="test-curl-7212-140156711137904" Jul 10 13:51:26.406346 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.406337 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" containerName="test-curl-7212-140156711137904" Jul 10 13:51:26.409277 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.409262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:26.415336 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.415316 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:26.457037 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.457011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjbs\" (UniqueName: \"kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs\") pod \"test-curl-7212-140156711138912\" (UID: \"e5e04b88-4418-41f5-9360-234bd9e8e4e8\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:26.558207 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.558183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjbs\" (UniqueName: \"kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs\") pod \"test-curl-7212-140156711138912\" (UID: \"e5e04b88-4418-41f5-9360-234bd9e8e4e8\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:26.565424 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.565403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjbs\" (UniqueName: \"kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs\") pod \"test-curl-7212-140156711138912\" (UID: \"e5e04b88-4418-41f5-9360-234bd9e8e4e8\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:26.719089 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.719057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:26.852446 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.852421 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:26.854297 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:26.854269 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5e04b88_4418_41f5_9360_234bd9e8e4e8.slice/crio-9307d75c9c3e5a7224ef820a3968c8ac57727f4b15e91983ba25a349b61efa6b WatchSource:0}: Error finding container 9307d75c9c3e5a7224ef820a3968c8ac57727f4b15e91983ba25a349b61efa6b: Status 404 returned error can't find the container with id 9307d75c9c3e5a7224ef820a3968c8ac57727f4b15e91983ba25a349b61efa6b Jul 10 13:51:26.968917 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:26.968890 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f936c5-1a99-4646-9afd-46c96d53bf6b" path="/var/lib/kubelet/pods/f3f936c5-1a99-4646-9afd-46c96d53bf6b/volumes" Jul 10 13:51:27.101264 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:27.101232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711138912" event={"ID":"e5e04b88-4418-41f5-9360-234bd9e8e4e8","Type":"ContainerStarted","Data":"9307d75c9c3e5a7224ef820a3968c8ac57727f4b15e91983ba25a349b61efa6b"} Jul 10 13:51:27.102499 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:27.102483 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711137904" Jul 10 13:51:27.102598 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:27.102484 2573 scope.go:117] "RemoveContainer" containerID="16ee5888b90c1306f48647bd334abfce130fb5b7a6c8bdecdbb8d4218bf75a78" Jul 10 13:51:27.382693 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:27.382662 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:51:28.108459 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:28.108426 2573 generic.go:356] "Generic (PLEG): container finished" podID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" containerID="cd495dc7d79dac49bcaadd2692a30eb9a431885bc0aed5307b347c1f5c5681be" exitCode=0 Jul 10 13:51:28.108619 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:28.108506 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711138912" event={"ID":"e5e04b88-4418-41f5-9360-234bd9e8e4e8","Type":"ContainerDied","Data":"cd495dc7d79dac49bcaadd2692a30eb9a431885bc0aed5307b347c1f5c5681be"} Jul 10 13:51:29.246419 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.246397 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.279156 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.279133 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156711138912_e5e04b88-4418-41f5-9360-234bd9e8e4e8/test-curl-7212-140156711138912/0.log" Jul 10 13:51:29.282239 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.282221 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qjbs\" (UniqueName: \"kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs\") pod \"e5e04b88-4418-41f5-9360-234bd9e8e4e8\" (UID: \"e5e04b88-4418-41f5-9360-234bd9e8e4e8\") " Jul 10 13:51:29.284142 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.284123 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs" (OuterVolumeSpecName: "kube-api-access-2qjbs") pod "e5e04b88-4418-41f5-9360-234bd9e8e4e8" (UID: "e5e04b88-4418-41f5-9360-234bd9e8e4e8"). InnerVolumeSpecName "kube-api-access-2qjbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:29.302279 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.302252 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:29.304973 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.304956 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:29.383061 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.383040 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjbs\" (UniqueName: \"kubernetes.io/projected/e5e04b88-4418-41f5-9360-234bd9e8e4e8-kube-api-access-2qjbs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.428469 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.428439 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:29.428880 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.428867 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" containerName="test-curl-7212-140156711138912" Jul 10 13:51:29.428921 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.428882 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" containerName="test-curl-7212-140156711138912" Jul 10 13:51:29.428952 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.428943 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" containerName="test-curl-7212-140156711138912" Jul 10 13:51:29.431826 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.431811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.436793 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.436767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:29.483427 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.483393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcgt\" (UniqueName: \"kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt\") pod \"test-curl-7212-140156711138912\" (UID: \"3165098b-f26b-4966-a37e-3c4da867e409\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.584615 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.584583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcgt\" (UniqueName: \"kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt\") pod \"test-curl-7212-140156711138912\" (UID: \"3165098b-f26b-4966-a37e-3c4da867e409\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.592315 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.592293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcgt\" (UniqueName: \"kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt\") pod \"test-curl-7212-140156711138912\" (UID: \"3165098b-f26b-4966-a37e-3c4da867e409\") " pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.742828 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.742760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:29.874070 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:29.874046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:29.876515 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:51:29.876492 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3165098b_f26b_4966_a37e_3c4da867e409.slice/crio-5cae80b8e7d5b5a7e243cf26b1bb177607f674c931f528d4a73ad99b3731327b WatchSource:0}: Error finding container 5cae80b8e7d5b5a7e243cf26b1bb177607f674c931f528d4a73ad99b3731327b: Status 404 returned error can't find the container with id 5cae80b8e7d5b5a7e243cf26b1bb177607f674c931f528d4a73ad99b3731327b Jul 10 13:51:30.120486 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:30.120395 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711138912" event={"ID":"3165098b-f26b-4966-a37e-3c4da867e409","Type":"ContainerStarted","Data":"5cae80b8e7d5b5a7e243cf26b1bb177607f674c931f528d4a73ad99b3731327b"} Jul 10 13:51:30.121644 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:30.121607 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:30.121791 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:30.121613 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9307d75c9c3e5a7224ef820a3968c8ac57727f4b15e91983ba25a349b61efa6b" Jul 10 13:51:30.135549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:30.135524 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7212-140156711138912" oldPodUID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" podUID="3165098b-f26b-4966-a37e-3c4da867e409" Jul 10 13:51:30.970055 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:30.970021 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e04b88-4418-41f5-9360-234bd9e8e4e8" path="/var/lib/kubelet/pods/e5e04b88-4418-41f5-9360-234bd9e8e4e8/volumes" Jul 10 13:51:31.130063 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:31.130027 2573 generic.go:356] "Generic (PLEG): container finished" podID="3165098b-f26b-4966-a37e-3c4da867e409" containerID="46f3c5b617d7fa6d7ea5b9aa34a9e53f3930b2e46ec792ee611c800d2913bcf4" exitCode=0 Jul 10 13:51:31.130228 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:31.130070 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7212-140156711138912" event={"ID":"3165098b-f26b-4966-a37e-3c4da867e409","Type":"ContainerDied","Data":"46f3c5b617d7fa6d7ea5b9aa34a9e53f3930b2e46ec792ee611c800d2913bcf4"} Jul 10 13:51:32.279091 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.279062 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:32.300478 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.300453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7212-140156711138912_3165098b-f26b-4966-a37e-3c4da867e409/test-curl-7212-140156711138912/0.log" Jul 10 13:51:32.308669 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.308624 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzcgt\" (UniqueName: \"kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt\") pod \"3165098b-f26b-4966-a37e-3c4da867e409\" (UID: \"3165098b-f26b-4966-a37e-3c4da867e409\") " Jul 10 13:51:32.310652 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.310604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt" (OuterVolumeSpecName: "kube-api-access-bzcgt") pod "3165098b-f26b-4966-a37e-3c4da867e409" (UID: "3165098b-f26b-4966-a37e-3c4da867e409"). InnerVolumeSpecName "kube-api-access-bzcgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:32.326059 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.326034 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:32.330491 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.330465 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7212-140156711138912"] Jul 10 13:51:32.409693 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.409610 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzcgt\" (UniqueName: \"kubernetes.io/projected/3165098b-f26b-4966-a37e-3c4da867e409-kube-api-access-bzcgt\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:32.969152 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:32.969120 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3165098b-f26b-4966-a37e-3c4da867e409" path="/var/lib/kubelet/pods/3165098b-f26b-4966-a37e-3c4da867e409/volumes" Jul 10 13:51:33.144095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.144063 2573 scope.go:117] "RemoveContainer" containerID="46f3c5b617d7fa6d7ea5b9aa34a9e53f3930b2e46ec792ee611c800d2913bcf4" Jul 10 13:51:33.144095 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.144082 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7212-140156711138912" Jul 10 13:51:33.254476 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.254398 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:51:33.254712 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.254688 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" podUID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" containerName="maas-api" containerID="cri-o://33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f" gracePeriod=30 Jul 10 13:51:33.729731 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.729707 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:51:33.821268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.821175 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls\") pod \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " Jul 10 13:51:33.821268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.821228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca\") pod \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " Jul 10 13:51:33.821268 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.821267 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6jv6\" (UniqueName: \"kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6\") pod \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\" (UID: \"9775a5a2-0f78-4e45-b78f-09f3eadfc33c\") " Jul 10 13:51:33.821541 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.821518 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9775a5a2-0f78-4e45-b78f-09f3eadfc33c" (UID: "9775a5a2-0f78-4e45-b78f-09f3eadfc33c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:51:33.823410 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.823385 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9775a5a2-0f78-4e45-b78f-09f3eadfc33c" (UID: "9775a5a2-0f78-4e45-b78f-09f3eadfc33c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:51:33.823535 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.823448 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6" (OuterVolumeSpecName: "kube-api-access-s6jv6") pod "9775a5a2-0f78-4e45-b78f-09f3eadfc33c" (UID: "9775a5a2-0f78-4e45-b78f-09f3eadfc33c"). InnerVolumeSpecName "kube-api-access-s6jv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:33.922386 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.922358 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-maas-api-tls\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:33.922495 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.922391 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-openshift-service-ca\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:33.922495 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:33.922405 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jv6\" (UniqueName: \"kubernetes.io/projected/9775a5a2-0f78-4e45-b78f-09f3eadfc33c-kube-api-access-s6jv6\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:34.150946 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.150914 2573 generic.go:356] "Generic (PLEG): container finished" podID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" containerID="33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f" exitCode=0 Jul 10 13:51:34.151139 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.150958 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" Jul 10 13:51:34.151139 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.150970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" event={"ID":"9775a5a2-0f78-4e45-b78f-09f3eadfc33c","Type":"ContainerDied","Data":"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f"} Jul 10 13:51:34.151139 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.151020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9" event={"ID":"9775a5a2-0f78-4e45-b78f-09f3eadfc33c","Type":"ContainerDied","Data":"49046fe05026e77f43aff8a224899d24e9d75d0df5e762ae35f3fb37218c8292"} Jul 10 13:51:34.151139 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.151048 2573 scope.go:117] "RemoveContainer" containerID="33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f" Jul 10 13:51:34.159766 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.159751 2573 scope.go:117] "RemoveContainer" containerID="33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f" Jul 10 13:51:34.160022 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:34.160003 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f\": container with ID starting with 33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f not found: ID does not exist" containerID="33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f" Jul 10 13:51:34.160085 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.160032 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f"} err="failed to get container status \"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f\": rpc error: code = NotFound desc = could not find container \"33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f\": container with ID starting with 33eccedf9fc4a8f7194a485d46a4d79cd8d51c9462d27d9d6ecfc3b711d2886f not found: ID does not exist" Jul 10 13:51:34.172344 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.172308 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:51:34.175776 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.175750 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90a164-694d8cb7ff-sphs9"] Jul 10 13:51:34.968651 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:34.968597 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" path="/var/lib/kubelet/pods/9775a5a2-0f78-4e45-b78f-09f3eadfc33c/volumes" Jul 10 13:51:39.386191 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:39.386160 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:51:39.386667 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:39.386420 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" containerID="cri-o://51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e" gracePeriod=30 Jul 10 13:51:39.393693 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:39.393666 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:51:43.859718 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:43.859674 2573 patch_prober.go:28] interesting pod/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 13:51:43.860179 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:43.859754 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:51:44.659734 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.659679 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:51:44.711030 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.710992 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711030 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711095 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711120 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqzm\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711182 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711216 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711286 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token\") pod \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\" (UID: \"ed85cd47-6028-4656-aafb-6ff99af1e1a2\") " Jul 10 13:51:44.711394 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711321 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:44.711788 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711529 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.711855 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.711789 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:44.712030 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.712002 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:51:44.712123 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.712084 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:44.712549 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.712524 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:44.713987 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.713961 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm" (OuterVolumeSpecName: "kube-api-access-8gqzm") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "kube-api-access-8gqzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:44.713987 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.713978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:51:44.714144 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.714049 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:44.714144 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.714126 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed85cd47-6028-4656-aafb-6ff99af1e1a2" (UID: "ed85cd47-6028-4656-aafb-6ff99af1e1a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:44.812332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812309 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-token\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812329 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-workload-certs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812332 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812337 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-envoy\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812345 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istiod-ca-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812353 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-credential-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812361 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqzm\" (UniqueName: \"kubernetes.io/projected/ed85cd47-6028-4656-aafb-6ff99af1e1a2-kube-api-access-8gqzm\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812369 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-data\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:44.812500 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:44.812377 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed85cd47-6028-4656-aafb-6ff99af1e1a2-istio-podinfo\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:45.208085 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.208047 2573 generic.go:356] "Generic (PLEG): container finished" podID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerID="51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e" exitCode=0 Jul 10 13:51:45.208470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.208103 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" Jul 10 13:51:45.208470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.208134 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerDied","Data":"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e"} Jul 10 13:51:45.208470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.208187 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn" event={"ID":"ed85cd47-6028-4656-aafb-6ff99af1e1a2","Type":"ContainerDied","Data":"d4d9adb3f50a933738e324b5b878d063deafb26ca8dde9b534e9684fa74e35ba"} Jul 10 13:51:45.208470 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.208210 2573 scope.go:117] "RemoveContainer" containerID="51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e" Jul 10 13:51:45.217257 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.217239 2573 scope.go:117] "RemoveContainer" containerID="eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2" Jul 10 13:51:45.225329 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.225312 2573 scope.go:117] "RemoveContainer" containerID="51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e" Jul 10 13:51:45.225611 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:45.225581 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e\": container with ID starting with 51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e not found: ID does not exist" containerID="51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e" Jul 10 13:51:45.225759 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.225621 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e"} err="failed to get container status \"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e\": rpc error: code = NotFound desc = could not find container \"51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e\": container with ID starting with 51ebcb1929d395c119de892d51617a73fb32db104a89d868e73133d47ee3fd7e not found: ID does not exist" Jul 10 13:51:45.225759 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.225660 2573 scope.go:117] "RemoveContainer" containerID="eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2" Jul 10 13:51:45.225927 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:45.225883 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2\": container with ID starting with eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2 not found: ID does not exist" containerID="eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2" Jul 10 13:51:45.225927 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.225912 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2"} err="failed to get container status \"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2\": rpc error: code = NotFound desc = could not find container \"eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2\": container with ID starting with eee1f18cd6052e55b946be045852f52c829a4c722ceb02d821bc8872b4e72ea2 not found: ID does not exist" Jul 10 13:51:45.226092 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.226075 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:51:45.232544 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:45.232525 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-90a164-openshift-default-8445c976f9-nnwcn"] Jul 10 13:51:46.626504 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:46.626470 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:51:46.626989 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:46.626786 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" containerID="cri-o://3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099" gracePeriod=30 Jul 10 13:51:46.969293 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:46.969215 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" path="/var/lib/kubelet/pods/ed85cd47-6028-4656-aafb-6ff99af1e1a2/volumes" Jul 10 13:51:49.582023 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:49.581992 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwxd"] Jul 10 13:51:51.377604 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.377567 2573 patch_prober.go:28] interesting pod/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 13:51:51.377984 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.377646 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:51:51.887416 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.887392 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:51:51.979396 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979299 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979396 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979396 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979392 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979416 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979440 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5ks5\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979544 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979586 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo\") pod \"71a9a067-2338-43d6-b8ea-963d49bcb547\" (UID: \"71a9a067-2338-43d6-b8ea-963d49bcb547\") " Jul 10 13:51:51.979744 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979707 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979777 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979867 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979971 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.979988 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-workload-certs\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:51.980081 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.980003 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-credential-socket\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:51.980442 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.980141 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data" (OuterVolumeSpecName: "istio-data") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:51.981834 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.981803 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:51.981961 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.981825 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5" (OuterVolumeSpecName: "kube-api-access-c5ks5") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "kube-api-access-c5ks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:51.981961 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.981843 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:51:51.981961 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:51.981883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token" (OuterVolumeSpecName: "istio-token") pod "71a9a067-2338-43d6-b8ea-963d49bcb547" (UID: "71a9a067-2338-43d6-b8ea-963d49bcb547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:52.080810 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080778 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-data\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.080810 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080803 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-envoy\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.081006 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080820 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a9a067-2338-43d6-b8ea-963d49bcb547-istiod-ca-cert\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.081006 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080830 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ks5\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-kube-api-access-c5ks5\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.081006 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080840 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-token\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.081006 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.080848 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a9a067-2338-43d6-b8ea-963d49bcb547-istio-podinfo\") on node \"ip-10-0-128-249.ec2.internal\" DevicePath \"\"" Jul 10 13:51:52.165285 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.165261 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wmdwn_e5089a22-2976-408a-9a13-08390cd73ee2/manager/0.log" Jul 10 13:51:52.246602 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.246527 2573 generic.go:356] "Generic (PLEG): container finished" podID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerID="3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099" exitCode=0 Jul 10 13:51:52.246602 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.246572 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" Jul 10 13:51:52.246784 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.246565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerDied","Data":"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099"} Jul 10 13:51:52.246784 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.246689 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h" event={"ID":"71a9a067-2338-43d6-b8ea-963d49bcb547","Type":"ContainerDied","Data":"ed1eb3b33eb636e7457e0eb05926da3d756bc6915558b0d657d9aaed1596a077"} Jul 10 13:51:52.246784 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.246713 2573 scope.go:117] "RemoveContainer" containerID="3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099" Jul 10 13:51:52.257707 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.257689 2573 scope.go:117] "RemoveContainer" containerID="3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7" Jul 10 13:51:52.267932 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.267910 2573 scope.go:117] "RemoveContainer" containerID="3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099" Jul 10 13:51:52.268169 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:52.268150 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099\": container with ID starting with 3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099 not found: ID does not exist" containerID="3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099" Jul 10 13:51:52.268236 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.268176 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099"} err="failed to get container status \"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099\": rpc error: code = NotFound desc = could not find container \"3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099\": container with ID starting with 3b3e096c4f5d52f11db12bc1019288042ebb8ad91e5f140dcf48538ab1c36099 not found: ID does not exist" Jul 10 13:51:52.268236 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.268198 2573 scope.go:117] "RemoveContainer" containerID="3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7" Jul 10 13:51:52.268421 ip-10-0-128-249 kubenswrapper[2573]: E0710 13:51:52.268395 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7\": container with ID starting with 3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7 not found: ID does not exist" containerID="3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7" Jul 10 13:51:52.268461 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.268426 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7"} err="failed to get container status \"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7\": rpc error: code = NotFound desc = could not find container \"3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7\": container with ID starting with 3b3f516c14906006b0e835e069eb77737273bcce9fc9404eb28beaf0905fe2e7 not found: ID does not exist" Jul 10 13:51:52.274182 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.274116 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:51:52.276456 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.276430 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-efd2ad-openshift-default-85f89fcc5-ql86h"] Jul 10 13:51:52.519522 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.519440 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5vrhw_2302ac8f-d09d-47bd-bd9b-be9aa5c943ae/server/0.log" Jul 10 13:51:52.639516 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.639490 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mczmh_473585ca-2dd7-41d2-95e0-ef7aabd71893/manager/0.log" Jul 10 13:51:52.880849 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.880822 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-j79kh_292b4d7e-3f17-431a-bfeb-eb7de0e0b920/copy-manifests/0.log" Jul 10 13:51:52.888126 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.888101 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-j79kh_292b4d7e-3f17-431a-bfeb-eb7de0e0b920/manager/0.log" Jul 10 13:51:52.968687 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:52.968657 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" path="/var/lib/kubelet/pods/71a9a067-2338-43d6-b8ea-963d49bcb547/volumes" Jul 10 13:51:54.852780 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:54.852701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dmbln_a93255e3-73ff-4aca-bf48-7aca5665f256/manager/0.log" Jul 10 13:51:54.969583 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:54.969551 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-92bss_91116b99-7e90-421e-97af-49ae06aa6b29/manager/0.log" Jul 10 13:51:55.082671 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:55.082625 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qvtss_d42c7116-492f-4386-b009-388cf862866c/kuadrant-console-plugin/0.log" Jul 10 13:51:55.208148 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:55.208117 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wxd5n_c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4/registry-server/0.log" Jul 10 13:51:55.337300 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:55.337271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qgx7n_7474bd25-817a-4dfe-9d72-e9c734120c49/manager/0.log" Jul 10 13:51:55.465294 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:55.465226 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vvwxd_d3e5f8c9-7133-45f1-93f0-07c9efa077a6/limitador/0.log" Jul 10 13:51:55.589326 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:55.589293 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l5t8r_e549be6b-ab7b-4a33-82f2-9bcfb91272ef/manager/0.log" Jul 10 13:51:56.171406 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:56.171379 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f6cf8c47-bpp2m_82b336ce-f12d-4467-a670-470d51d57edb/kube-auth-proxy/0.log" Jul 10 13:51:56.741266 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:51:56.741241 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755b7d5488-4vbsm_ca3745bd-546f-4cc6-8edf-672d5511f623/router/0.log" Jul 10 13:52:01.624284 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624246 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k76w9/must-gather-94md6"] Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3165098b-f26b-4966-a37e-3c4da867e409" containerName="test-curl-7212-140156711138912" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624603 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3165098b-f26b-4966-a37e-3c4da867e409" containerName="test-curl-7212-140156711138912" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624614 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" containerName="maas-api" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624619 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" containerName="maas-api" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624655 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624661 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624667 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.624677 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624674 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624687 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624692 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624704 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624709 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624766 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624773 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624785 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a9a067-2338-43d6-b8ea-963d49bcb547" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624791 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed85cd47-6028-4656-aafb-6ff99af1e1a2" containerName="istio-proxy" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624796 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3165098b-f26b-4966-a37e-3c4da867e409" containerName="test-curl-7212-140156711138912" Jul 10 13:52:01.625012 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.624802 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9775a5a2-0f78-4e45-b78f-09f3eadfc33c" containerName="maas-api" Jul 10 13:52:01.629551 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.629525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.631981 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.631956 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k76w9\"/\"openshift-service-ca.crt\"" Jul 10 13:52:01.632105 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.631964 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k76w9\"/\"kube-root-ca.crt\"" Jul 10 13:52:01.632780 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.632765 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k76w9\"/\"default-dockercfg-wps8p\"" Jul 10 13:52:01.636754 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.636733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/must-gather-94md6"] Jul 10 13:52:01.763990 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.763963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gnnp\" (UniqueName: \"kubernetes.io/projected/ef22aabd-3870-48f6-9849-de7dd7d2c57e-kube-api-access-4gnnp\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.764111 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.764010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef22aabd-3870-48f6-9849-de7dd7d2c57e-must-gather-output\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.864668 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.864616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef22aabd-3870-48f6-9849-de7dd7d2c57e-must-gather-output\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.864800 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.864734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnnp\" (UniqueName: \"kubernetes.io/projected/ef22aabd-3870-48f6-9849-de7dd7d2c57e-kube-api-access-4gnnp\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.864895 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.864880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ef22aabd-3870-48f6-9849-de7dd7d2c57e-must-gather-output\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.880044 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.879989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnnp\" (UniqueName: \"kubernetes.io/projected/ef22aabd-3870-48f6-9849-de7dd7d2c57e-kube-api-access-4gnnp\") pod \"must-gather-94md6\" (UID: \"ef22aabd-3870-48f6-9849-de7dd7d2c57e\") " pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:01.940907 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:01.940887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/must-gather-94md6" Jul 10 13:52:02.081915 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:02.081895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/must-gather-94md6"] Jul 10 13:52:02.084429 ip-10-0-128-249 kubenswrapper[2573]: W0710 13:52:02.084398 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef22aabd_3870_48f6_9849_de7dd7d2c57e.slice/crio-59077bdb68d12a8b23abe5bfaa8bcdcd450065f444bc7ff1edc23ffbebf86a52 WatchSource:0}: Error finding container 59077bdb68d12a8b23abe5bfaa8bcdcd450065f444bc7ff1edc23ffbebf86a52: Status 404 returned error can't find the container with id 59077bdb68d12a8b23abe5bfaa8bcdcd450065f444bc7ff1edc23ffbebf86a52 Jul 10 13:52:02.298824 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:02.298728 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/must-gather-94md6" event={"ID":"ef22aabd-3870-48f6-9849-de7dd7d2c57e","Type":"ContainerStarted","Data":"59077bdb68d12a8b23abe5bfaa8bcdcd450065f444bc7ff1edc23ffbebf86a52"} Jul 10 13:52:03.305901 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:03.305851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/must-gather-94md6" event={"ID":"ef22aabd-3870-48f6-9849-de7dd7d2c57e","Type":"ContainerStarted","Data":"8aecca69a5680c815cbe3a092ba410933a6d9db36b30562eab4bbf4a170595d1"} Jul 10 13:52:03.305901 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:03.305899 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/must-gather-94md6" event={"ID":"ef22aabd-3870-48f6-9849-de7dd7d2c57e","Type":"ContainerStarted","Data":"0e994ed1f910a4421fa0683b7a61e1a40f2e7ad1f7bc9a08d81eb25b45e626b3"} Jul 10 13:52:03.322715 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:03.322653 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k76w9/must-gather-94md6" podStartSLOduration=1.612258052 podStartE2EDuration="2.322606235s" totalImagesPullingTime="710.348183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:52:03.319933731 +0000 UTC m=+2520.852536031" watchObservedRunningTime="2026-07-10 13:52:03.322606235 +0000 UTC m=+2520.855208535" Jul 10 13:52:04.412176 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:04.412141 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-28kx4_36ab361d-d9c8-4d97-bf03-4393b79c32f8/global-pull-secret-syncer/0.log" Jul 10 13:52:04.643576 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:04.643549 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xt2hl_f9e7b8a3-80e5-4c3a-b613-ae8f57e8975b/konnectivity-agent/0.log" Jul 10 13:52:04.671326 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:04.671245 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-249.ec2.internal_6184f72f327664492ecaec42e6fc927e/haproxy/0.log" Jul 10 13:52:10.007769 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.007735 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dmbln_a93255e3-73ff-4aca-bf48-7aca5665f256/manager/0.log" Jul 10 13:52:10.041771 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.041733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-92bss_91116b99-7e90-421e-97af-49ae06aa6b29/manager/0.log" Jul 10 13:52:10.069257 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.069223 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qvtss_d42c7116-492f-4386-b009-388cf862866c/kuadrant-console-plugin/0.log" Jul 10 13:52:10.115510 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.115470 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wxd5n_c16cea15-9dfd-4e0e-87d0-bfce4ed2e0d4/registry-server/0.log" Jul 10 13:52:10.278704 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.277079 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qgx7n_7474bd25-817a-4dfe-9d72-e9c734120c49/manager/0.log" Jul 10 13:52:10.324830 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.324799 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vvwxd_d3e5f8c9-7133-45f1-93f0-07c9efa077a6/limitador/0.log" Jul 10 13:52:10.441945 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:10.441912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l5t8r_e549be6b-ab7b-4a33-82f2-9bcfb91272ef/manager/0.log" Jul 10 13:52:12.244373 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:12.244288 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-jxjvk_d6a98def-23d5-404f-8429-18347a24e8b9/cluster-monitoring-operator/0.log" Jul 10 13:52:12.647229 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:12.647194 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcqmr_642eb80f-115d-4e22-9761-b44da23c69af/node-exporter/0.log" Jul 10 13:52:12.674227 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:12.674201 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcqmr_642eb80f-115d-4e22-9761-b44da23c69af/kube-rbac-proxy/0.log" Jul 10 13:52:12.698529 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:12.698501 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcqmr_642eb80f-115d-4e22-9761-b44da23c69af/init-textfile/0.log" Jul 10 13:52:14.525597 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.525555 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-qk7gx_b364a518-6d8e-4b67-b242-d64942c29b50/networking-console-plugin/0.log" Jul 10 13:52:14.553303 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.553256 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d"] Jul 10 13:52:14.558539 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.558515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.565790 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.565034 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d"] Jul 10 13:52:14.615449 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.615412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-proc\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.615772 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.615748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-podres\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.615983 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.615965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnpkz\" (UniqueName: \"kubernetes.io/projected/e30aaed6-c7d9-4d38-bf60-333c38343145-kube-api-access-mnpkz\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.616166 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.616152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-lib-modules\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.616371 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.616357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-sys\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717832 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-proc\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-podres\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnpkz\" (UniqueName: \"kubernetes.io/projected/e30aaed6-c7d9-4d38-bf60-333c38343145-kube-api-access-mnpkz\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-lib-modules\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-proc\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-sys\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.717997 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.717992 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-podres\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.718270 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.718015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-sys\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.718270 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.718113 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30aaed6-c7d9-4d38-bf60-333c38343145-lib-modules\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.725550 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.725527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnpkz\" (UniqueName: \"kubernetes.io/projected/e30aaed6-c7d9-4d38-bf60-333c38343145-kube-api-access-mnpkz\") pod \"perf-node-gather-daemonset-rkc7d\" (UID: \"e30aaed6-c7d9-4d38-bf60-333c38343145\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:14.878207 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:14.878173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:15.069483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.069457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d"] Jul 10 13:52:15.173777 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.173745 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/1.log" Jul 10 13:52:15.183419 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.183395 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jdskh_509c9cde-d447-4dd4-b153-37eb06725a6a/console-operator/2.log" Jul 10 13:52:15.398108 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.398023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" event={"ID":"e30aaed6-c7d9-4d38-bf60-333c38343145","Type":"ContainerStarted","Data":"2b17468b5ebc093a9e2799cec48c62b251b4d25dadf16f30a605bf5aa2050e8e"} Jul 10 13:52:15.398277 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.398113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" event={"ID":"e30aaed6-c7d9-4d38-bf60-333c38343145","Type":"ContainerStarted","Data":"649b7ebc148a05e3b9766ab7c58dffad2e2157eea18308c10a4faa37f0d27282"} Jul 10 13:52:15.398277 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.398160 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:15.417064 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.417003 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" podStartSLOduration=1.4169796909999999 podStartE2EDuration="1.416979691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:52:15.41398707 +0000 UTC m=+2532.946589370" watchObservedRunningTime="2026-07-10 13:52:15.416979691 +0000 UTC m=+2532.949581991" Jul 10 13:52:15.679952 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:15.679879 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-tlxnd_228f3243-96f6-49a8-935a-6a7bf9d04c84/download-server/0.log" Jul 10 13:52:16.273141 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:16.273100 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-vvthg_a41dee19-9297-424c-a35e-41dbf2f882c7/volume-data-source-validator/0.log" Jul 10 13:52:17.352297 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:17.352267 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjh5l_afc44162-1c8b-427f-a7fb-ac64c02eb3af/dns/0.log" Jul 10 13:52:17.379918 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:17.379891 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjh5l_afc44162-1c8b-427f-a7fb-ac64c02eb3af/kube-rbac-proxy/0.log" Jul 10 13:52:17.530171 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:17.530145 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zb9b_47874066-db3d-4c40-9c31-e4ee15f49de4/dns-node-resolver/0.log" Jul 10 13:52:18.163807 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:18.163767 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bp48_c526be69-1738-4d64-8989-bb9c151b0ce6/node-ca/0.log" Jul 10 13:52:19.450113 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:19.450083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f6cf8c47-bpp2m_82b336ce-f12d-4467-a670-470d51d57edb/kube-auth-proxy/0.log" Jul 10 13:52:19.716757 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:19.716662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755b7d5488-4vbsm_ca3745bd-546f-4cc6-8edf-672d5511f623/router/0.log" Jul 10 13:52:20.484961 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:20.484929 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hhpjb_e571d6b8-14f6-4e5d-906b-819d485b0117/serve-healthcheck-canary/0.log" Jul 10 13:52:21.046493 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.046464 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-24fqk_b4e79848-1a24-43f0-bee6-1590702cfaca/insights-operator/0.log" Jul 10 13:52:21.047975 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.047952 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-24fqk_b4e79848-1a24-43f0-bee6-1590702cfaca/insights-operator/1.log" Jul 10 13:52:21.245774 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.245748 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmhkk_44f5d802-1f73-4843-ac3e-c7985f017eb6/kube-rbac-proxy/0.log" Jul 10 13:52:21.271614 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.271590 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmhkk_44f5d802-1f73-4843-ac3e-c7985f017eb6/exporter/0.log" Jul 10 13:52:21.303529 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.303443 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmhkk_44f5d802-1f73-4843-ac3e-c7985f017eb6/extractor/0.log" Jul 10 13:52:21.416388 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:21.416356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-rkc7d" Jul 10 13:52:23.638194 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:23.638137 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wmdwn_e5089a22-2976-408a-9a13-08390cd73ee2/manager/0.log" Jul 10 13:52:23.884306 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:23.884285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5vrhw_2302ac8f-d09d-47bd-bd9b-be9aa5c943ae/server/0.log" Jul 10 13:52:24.002891 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:24.002859 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mczmh_473585ca-2dd7-41d2-95e0-ef7aabd71893/manager/0.log" Jul 10 13:52:24.209819 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:24.209738 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-j79kh_292b4d7e-3f17-431a-bfeb-eb7de0e0b920/manager/0.log" Jul 10 13:52:24.239132 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:24.239102 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-j79kh_292b4d7e-3f17-431a-bfeb-eb7de0e0b920/copy-manifests/0.log" Jul 10 13:52:25.322543 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:25.322512 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f6cf8c47-bpp2m_82b336ce-f12d-4467-a670-470d51d57edb/kube-auth-proxy/0.log" Jul 10 13:52:25.480794 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:25.480762 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755b7d5488-4vbsm_ca3745bd-546f-4cc6-8edf-672d5511f623/router/0.log" Jul 10 13:52:26.882471 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:26.882440 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f6cf8c47-bpp2m_82b336ce-f12d-4467-a670-470d51d57edb/kube-auth-proxy/0.log" Jul 10 13:52:27.040712 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:27.040621 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-755b7d5488-4vbsm_ca3745bd-546f-4cc6-8edf-672d5511f623/router/0.log" Jul 10 13:52:30.554760 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:30.554733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gq7bz_a2b4853b-41b6-4f9d-9136-17468cb36fa3/migrator/0.log" Jul 10 13:52:30.575173 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:30.575147 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gq7bz_a2b4853b-41b6-4f9d-9136-17468cb36fa3/graceful-termination/0.log" Jul 10 13:52:30.912374 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:30.912340 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zmnmf_e2beeec8-8f39-4756-8e53-4403d5c4d19c/kube-storage-version-migrator-operator/1.log" Jul 10 13:52:30.914112 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:30.914056 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zmnmf_e2beeec8-8f39-4756-8e53-4403d5c4d19c/kube-storage-version-migrator-operator/0.log" Jul 10 13:52:31.939412 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:31.939380 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/kube-multus-additional-cni-plugins/0.log" Jul 10 13:52:31.960483 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:31.960461 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/egress-router-binary-copy/0.log" Jul 10 13:52:31.980821 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:31.980800 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/cni-plugins/0.log" Jul 10 13:52:32.001442 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:32.001427 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/bond-cni-plugin/0.log" Jul 10 13:52:32.022555 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:32.022531 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/routeoverride-cni/0.log" Jul 10 13:52:32.044879 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:32.044851 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/whereabouts-cni-bincopy/0.log" Jul 10 13:52:32.069537 ip-10-0-128-249 kubenswrapper[2573]: I0710 13:52:32.069510 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qsqd_82bbaed0-7855-4587-b7c4-2d4b15ce23ea/whereabouts-cni/0.log"